{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  &#x1F4D1; **作业 5: 机器翻译**\n",
    "- 英译中 (繁体)\n",
    "  - 输入: an English sentence (比如: tom is a student .)\n",
    "  - 输出: the Chinese translation  (比如: 湯姆 是 個 學生 。)\n",
    "\n",
    "- <font color=darkred><b>***TODO***:</font></b>\n",
    "    - <font color=darkred>训练一个简单的 RNN seq2seq模型，实现翻译功能</font>\n",
    "    - <font color=darkred>改用transformer模型提升模型表现</font>\n",
    "    - <font color=darkred>Apply Back-translation to furthur boost performance</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 导入包\n",
    "\n",
    "安装： \n",
    "\n",
    "- **editdistance**： 快速实现编辑距离（Levenshtein距离）。  \n",
    "- **sacrebleu**： 计算bleu的库, 可以查看下[知乎: BLEU指标及评测脚本使用的一些误解](https://zhuanlan.zhihu.com/p/404381278)\n",
    "- **sacremoses**: 使用Python实现了Moses的tokenizer, truecaser以及normalizer功能，使用起来比较方便[官方Github（有示例）](https://github.com/alvations/sacremoses)\n",
    "- **sentencepiece**： 由谷歌将一些词-语言模型相关的论文进行复现，开发了一个开源工具——训练自己领域的sentencepiece模型，该模型可以代替预训练模型(BERT,XLNET)中词表的作用，可以参考[sentencepiece原理与实践](https://zhuanlan.zhihu.com/p/159200073)\n",
    "- **wandb**: 是Weights & Biases的缩写，这款工具能够帮助跟踪你的机器学习项目。它能够自动记录模型训练过程中的超参数和输出指标，然后可视化和比较结果，并快速与同事共享结果。[官方文档：quickstart](https://docs.wandb.ai/v/zh-hans/quickstart)\n",
    "- **fairseq**: 一个用PyTorch编写的序列建模工具包，它允许研究人员和开发人员训练用于翻译、摘要、语言建模和其他文本生成任务的自定义模型。[fairseq官方文档](https://fairseq.readthedocs.io/en/latest/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:37:28.555757Z",
     "iopub.status.busy": "2023-03-12T11:37:28.555327Z",
     "iopub.status.idle": "2023-03-12T11:37:41.979992Z",
     "shell.execute_reply": "2023-03-12T11:37:41.978771Z",
     "shell.execute_reply.started": "2023-03-12T11:37:28.555671Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting editdistance\n",
      "  Downloading editdistance-0.6.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (282 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m282.6/282.6 kB\u001b[0m \u001b[31m16.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting sacrebleu\n",
      "  Downloading sacrebleu-2.3.1-py3-none-any.whl (118 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m118.9/118.9 kB\u001b[0m \u001b[31m11.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: sacremoses in /opt/conda/lib/python3.7/site-packages (0.0.53)\n",
      "Requirement already satisfied: sentencepiece in /opt/conda/lib/python3.7/site-packages (0.1.97)\n",
      "Requirement already satisfied: wandb in /opt/conda/lib/python3.7/site-packages (0.12.21)\n",
      "Requirement already satisfied: regex in /opt/conda/lib/python3.7/site-packages (from sacrebleu) (2021.11.10)\n",
      "Requirement already satisfied: lxml in /opt/conda/lib/python3.7/site-packages (from sacrebleu) (4.9.2)\n",
      "Requirement already satisfied: tabulate>=0.8.9 in /opt/conda/lib/python3.7/site-packages (from sacrebleu) (0.9.0)\n",
      "Requirement already satisfied: colorama in /opt/conda/lib/python3.7/site-packages (from sacrebleu) (0.4.6)\n",
      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.7/site-packages (from sacrebleu) (1.21.6)\n",
      "Requirement already satisfied: portalocker in /opt/conda/lib/python3.7/site-packages (from sacrebleu) (2.7.0)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from sacremoses) (4.64.0)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from sacremoses) (1.15.0)\n",
      "Requirement already satisfied: joblib in /opt/conda/lib/python3.7/site-packages (from sacremoses) (1.0.1)\n",
      "Requirement already satisfied: click in /opt/conda/lib/python3.7/site-packages (from sacremoses) (8.1.3)\n",
      "Requirement already satisfied: psutil>=5.0.0 in /opt/conda/lib/python3.7/site-packages (from wandb) (5.9.1)\n",
      "Requirement already satisfied: pathtools in /opt/conda/lib/python3.7/site-packages (from wandb) (0.1.2)\n",
      "Requirement already satisfied: requests<3,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from wandb) (2.28.1)\n",
      "Requirement already satisfied: setproctitle in /opt/conda/lib/python3.7/site-packages (from wandb) (1.3.2)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from wandb) (59.8.0)\n",
      "Requirement already satisfied: GitPython>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from wandb) (3.1.27)\n",
      "Requirement already satisfied: shortuuid>=0.5.0 in /opt/conda/lib/python3.7/site-packages (from wandb) (1.0.11)\n",
      "Requirement already satisfied: docker-pycreds>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from wandb) (0.4.0)\n",
      "Requirement already satisfied: protobuf<4.0dev,>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from wandb) (3.20.3)\n",
      "Requirement already satisfied: promise<3,>=2.0 in /opt/conda/lib/python3.7/site-packages (from wandb) (2.3)\n",
      "Requirement already satisfied: PyYAML in /opt/conda/lib/python3.7/site-packages (from wandb) (6.0)\n",
      "Requirement already satisfied: sentry-sdk>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from wandb) (1.13.0)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from click->sacremoses) (6.0.0)\n",
      "Requirement already satisfied: gitdb<5,>=4.0.1 in /opt/conda/lib/python3.7/site-packages (from GitPython>=1.0.0->wandb) (4.0.9)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.7/site-packages (from GitPython>=1.0.0->wandb) (4.1.1)\n",
      "Requirement already satisfied: charset-normalizer<3,>=2 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.0.0->wandb) (2.1.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.0.0->wandb) (3.3)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.0.0->wandb) (1.26.14)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.0.0->wandb) (2022.12.7)\n",
      "Requirement already satisfied: smmap<6,>=3.0.1 in /opt/conda/lib/python3.7/site-packages (from gitdb<5,>=4.0.1->GitPython>=1.0.0->wandb) (3.0.5)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->click->sacremoses) (3.8.0)\n",
      "Installing collected packages: sacrebleu, editdistance\n",
      "Successfully installed editdistance-0.6.2 sacrebleu-2.3.1\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install  editdistance  sacrebleu sacremoses sentencepiece wandb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:37:41.982968Z",
     "iopub.status.busy": "2023-03-12T11:37:41.982570Z",
     "iopub.status.idle": "2023-03-12T11:38:44.332782Z",
     "shell.execute_reply": "2023-03-12T11:38:44.331589Z",
     "shell.execute_reply.started": "2023-03-12T11:37:41.982924Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'fairseq'...\n",
      "remote: Enumerating objects: 34534, done.\u001b[K\n",
      "remote: Total 34534 (delta 0), reused 0 (delta 0), pack-reused 34534\u001b[K\n",
      "Receiving objects: 100% (34534/34534), 24.06 MiB | 14.24 MiB/s, done.\n",
      "Resolving deltas: 100% (25109/25109), done.\n",
      "Note: switching to '9a1c497'.\n",
      "\n",
      "You are in 'detached HEAD' state. You can look around, make experimental\n",
      "changes and commit them, and you can discard any commits you make in this\n",
      "state without impacting any branches by switching back to a branch.\n",
      "\n",
      "If you want to create a new branch to retain commits you create, you may\n",
      "do so (now or later) by using -c with the switch command. Example:\n",
      "\n",
      "  git switch -c <new-branch-name>\n",
      "\n",
      "Or undo this operation with:\n",
      "\n",
      "  git switch -\n",
      "\n",
      "Turn off this advice by setting config variable advice.detachedHead to false\n",
      "\n",
      "HEAD is now at 9a1c4970 Make Hydra logging work with DDP (#1568)\n",
      "Processing ./fairseq\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Installing backend dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting hydra-core<1.1\n",
      "  Downloading hydra_core-1.0.7-py3-none-any.whl (123 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m123.8/123.8 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: cffi in /opt/conda/lib/python3.7/site-packages (from fairseq==1.0.0a0+9a1c497) (1.15.0)\n",
      "Requirement already satisfied: cython in /opt/conda/lib/python3.7/site-packages (from fairseq==1.0.0a0+9a1c497) (0.29.33)\n",
      "Requirement already satisfied: torch in /opt/conda/lib/python3.7/site-packages (from fairseq==1.0.0a0+9a1c497) (1.11.0)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from fairseq==1.0.0a0+9a1c497) (4.64.0)\n",
      "Requirement already satisfied: sacrebleu>=1.4.12 in /opt/conda/lib/python3.7/site-packages (from fairseq==1.0.0a0+9a1c497) (2.3.1)\n",
      "Requirement already satisfied: regex in /opt/conda/lib/python3.7/site-packages (from fairseq==1.0.0a0+9a1c497) (2021.11.10)\n",
      "Collecting omegaconf<2.1\n",
      "  Downloading omegaconf-2.0.6-py3-none-any.whl (36 kB)\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from fairseq==1.0.0a0+9a1c497) (1.21.6)\n",
      "Collecting antlr4-python3-runtime==4.8\n",
      "  Downloading antlr4-python3-runtime-4.8.tar.gz (112 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m112.4/112.4 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: importlib-resources in /opt/conda/lib/python3.7/site-packages (from hydra-core<1.1->fairseq==1.0.0a0+9a1c497) (5.10.2)\n",
      "Requirement already satisfied: PyYAML>=5.1.* in /opt/conda/lib/python3.7/site-packages (from omegaconf<2.1->fairseq==1.0.0a0+9a1c497) (6.0)\n",
      "Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from omegaconf<2.1->fairseq==1.0.0a0+9a1c497) (4.1.1)\n",
      "Requirement already satisfied: portalocker in /opt/conda/lib/python3.7/site-packages (from sacrebleu>=1.4.12->fairseq==1.0.0a0+9a1c497) (2.7.0)\n",
      "Requirement already satisfied: tabulate>=0.8.9 in /opt/conda/lib/python3.7/site-packages (from sacrebleu>=1.4.12->fairseq==1.0.0a0+9a1c497) (0.9.0)\n",
      "Requirement already satisfied: colorama in /opt/conda/lib/python3.7/site-packages (from sacrebleu>=1.4.12->fairseq==1.0.0a0+9a1c497) (0.4.6)\n",
      "Requirement already satisfied: lxml in /opt/conda/lib/python3.7/site-packages (from sacrebleu>=1.4.12->fairseq==1.0.0a0+9a1c497) (4.9.2)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi->fairseq==1.0.0a0+9a1c497) (2.21)\n",
      "Requirement already satisfied: zipp>=3.1.0 in /opt/conda/lib/python3.7/site-packages (from importlib-resources->hydra-core<1.1->fairseq==1.0.0a0+9a1c497) (3.8.0)\n",
      "Building wheels for collected packages: fairseq, antlr4-python3-runtime\n",
      "  Building wheel for fairseq (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for fairseq: filename=fairseq-1.0.0a0+9a1c497-cp37-cp37m-linux_x86_64.whl size=2039145 sha256=d79695c06ec03497bac689b1b09768369cfb4fb4738def0a626cf1fb8406752f\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-eovu564a/wheels/c8/ca/fd/f57b5450f90d8ce5f5b0bca2d8533572e2be617a079c9b2718\n",
      "  Building wheel for antlr4-python3-runtime (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.8-py3-none-any.whl size=141230 sha256=1e4473a1d597576856e1d12ee0d7b117070dd6aaf33f6fd1baf5605ca21eaa91\n",
      "  Stored in directory: /root/.cache/pip/wheels/ca/33/b7/336836125fc9bb4ceaa4376d8abca10ca8bc84ddc824baea6c\n",
      "Successfully built fairseq antlr4-python3-runtime\n",
      "Installing collected packages: antlr4-python3-runtime, omegaconf, hydra-core, fairseq\n",
      "Successfully installed antlr4-python3-runtime-4.8 fairseq-1.0.0a0+9a1c497 hydra-core-1.0.7 omegaconf-2.0.6\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# https://fairseq.readthedocs.io/en/latest/\n",
    "!git clone https://github.com/pytorch/fairseq.git\n",
    "!cd fairseq && git checkout 9a1c497\n",
    "!pip install --upgrade ./fairseq/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2023-03-12T11:38:44.336745Z",
     "iopub.status.busy": "2023-03-12T11:38:44.335763Z",
     "iopub.status.idle": "2023-03-12T11:38:47.321146Z",
     "shell.execute_reply": "2023-03-12T11:38:47.320118Z",
     "shell.execute_reply.started": "2023-03-12T11:38:44.336708Z"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "import pdb\n",
    "import pprint\n",
    "import logging\n",
    "import random\n",
    "import re\n",
    "\n",
    "import numpy as np\n",
    "# from tqdm.auto import tqdm \n",
    "from tqdm import tqdm # auto下载之后会显示不出来\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "from torch.utils.data import DataLoader, Dataset\n",
    "from pathlib import Path\n",
    "from argparse import Namespace\n",
    "from fairseq import utils\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 一些功能函数\n",
    "\n",
    "一些重要的方法（随机种子设置）  \n",
    "<b>这部分可以不做修改</b>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:38:47.324832Z",
     "iopub.status.busy": "2023-03-12T11:38:47.323776Z",
     "iopub.status.idle": "2023-03-12T11:38:47.337512Z",
     "shell.execute_reply": "2023-03-12T11:38:47.336382Z",
     "shell.execute_reply.started": "2023-03-12T11:38:47.324790Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Set env random_seed = 73\n"
     ]
    }
   ],
   "source": [
    "def all_seed(seed=6666):\n",
    "    np.random.seed(seed)\n",
    "    random.seed(seed)\n",
    "    # CPU\n",
    "    torch.manual_seed(seed)\n",
    "    # GPU\n",
    "    if torch.cuda.is_available():\n",
    "        torch.cuda.manual_seed_all(seed)\n",
    "        torch.cuda.manual_seed(seed)\n",
    "    # python全局\n",
    "    os.environ['PYTHONHASHSEED'] = str(seed)\n",
    "    # cudnn\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "    torch.backends.cudnn.benchmark = False\n",
    "    torch.backends.cudnn.enabled = False\n",
    "    print(f'Set env random_seed = {seed}')\n",
    "    \n",
    "    \n",
    "all_seed(73)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-29T06:52:13.391243Z",
     "iopub.status.busy": "2023-01-29T06:52:13.390759Z",
     "iopub.status.idle": "2023-01-29T06:52:13.400509Z",
     "shell.execute_reply": "2023-01-29T06:52:13.398636Z",
     "shell.execute_reply.started": "2023-01-29T06:52:13.391207Z"
    }
   },
   "source": [
    "# 数据集下载\n",
    "\n",
    "## En-Zh Bilingual Parallel Corpus  （英-中 平行语料）\n",
    "* [TED2020](#reimers-2020-multilingual-sentence-bert)\n",
    "    - Raw: 398,066 (sentences)   \n",
    "    - Processed: 393,980 (sentences)\n",
    "\n",
    "## 测试数据\n",
    "- 大小: 4,000 (句子)\n",
    "- **中文翻译并未提供。 提供的（.zh）文件是伪翻译，每一行都是一个'。'**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:38:47.339667Z",
     "iopub.status.busy": "2023-03-12T11:38:47.339166Z",
     "iopub.status.idle": "2023-03-12T11:39:04.236077Z",
     "shell.execute_reply": "2023-03-12T11:39:04.234733Z",
     "shell.execute_reply.started": "2023-03-12T11:38:47.339631Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-03-12 11:38:48--  https://github.com/yuhsinchan/ML2022-HW5Dataset/releases/download/v1.0.2/ted2020.tgz\n",
      "Resolving github.com (github.com)... 20.248.137.48\n",
      "Connecting to github.com (github.com)|20.248.137.48|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/465173291/b6199fd0-8de5-4614-80e9-fc1f461fa257?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230312%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230312T113848Z&X-Amz-Expires=300&X-Amz-Signature=3194f160e9515be822c2cd7d3e6b65bef4c569e8916b48385e2b194c9570b09a&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=465173291&response-content-disposition=attachment%3B%20filename%3Dted2020.tgz&response-content-type=application%2Foctet-stream [following]\n",
      "--2023-03-12 11:38:48--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/465173291/b6199fd0-8de5-4614-80e9-fc1f461fa257?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230312%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230312T113848Z&X-Amz-Expires=300&X-Amz-Signature=3194f160e9515be822c2cd7d3e6b65bef4c569e8916b48385e2b194c9570b09a&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=465173291&response-content-disposition=attachment%3B%20filename%3Dted2020.tgz&response-content-type=application%2Foctet-stream\n",
      "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
      "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 28601955 (27M) [application/octet-stream]\n",
      "Saving to: ‘/kaggle/working/DATA/rawdata/ted2020/ted2020.tgz’\n",
      "\n",
      "/kaggle/working/DAT 100%[===================>]  27.28M  6.69MB/s    in 4.3s    \n",
      "\n",
      "2023-03-12 11:38:53 (6.38 MB/s) - ‘/kaggle/working/DATA/rawdata/ted2020/ted2020.tgz’ saved [28601955/28601955]\n",
      "\n",
      "raw.en\n",
      "raw.zh\n",
      "--2023-03-12 11:38:56--  https://github.com/yuhsinchan/ML2022-HW5Dataset/releases/download/v1.0.2/test.tgz\n",
      "Resolving github.com (github.com)... 20.248.137.48\n",
      "Connecting to github.com (github.com)|20.248.137.48|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/465173291/0b97c817-32b7-433f-a7f5-036c8603991f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230312%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230312T113856Z&X-Amz-Expires=300&X-Amz-Signature=707d99519c16aede4e9e52c63bb69e8f2362de1e5a957f32998289320e2b590d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=465173291&response-content-disposition=attachment%3B%20filename%3Dtest.tgz&response-content-type=application%2Foctet-stream [following]\n",
      "--2023-03-12 11:38:56--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/465173291/0b97c817-32b7-433f-a7f5-036c8603991f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230312%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230312T113856Z&X-Amz-Expires=300&X-Amz-Signature=707d99519c16aede4e9e52c63bb69e8f2362de1e5a957f32998289320e2b590d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=465173291&response-content-disposition=attachment%3B%20filename%3Dtest.tgz&response-content-type=application%2Foctet-stream\n",
      "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
      "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 399360 (390K) [application/octet-stream]\n",
      "Saving to: ‘/kaggle/working/DATA/rawdata/ted2020/test.tgz’\n",
      "\n",
      "/kaggle/working/DAT 100%[===================>] 390.00K   391KB/s    in 1.0s    \n",
      "\n",
      "2023-03-12 11:38:58 (391 KB/s) - ‘/kaggle/working/DATA/rawdata/ted2020/test.tgz’ saved [399360/399360]\n",
      "\n",
      "test/\n",
      "test/test.zh\n",
      "test/test.en\n"
     ]
    }
   ],
   "source": [
    "data_dir = './DATA/rawdata'\n",
    "dataset_name = 'ted2020'\n",
    "urls = (\n",
    "    \"https://github.com/yuhsinchan/ML2022-HW5Dataset/releases/download/v1.0.2/ted2020.tgz\",\n",
    "    \"https://github.com/yuhsinchan/ML2022-HW5Dataset/releases/download/v1.0.2/test.tgz\",\n",
    ")\n",
    "file_names = (\n",
    "    'ted2020.tgz', # train & dev\n",
    "    'test.tgz', # test\n",
    ")\n",
    "prefix = Path(data_dir).absolute() / dataset_name\n",
    "\n",
    "prefix.mkdir(parents=True, exist_ok=True)\n",
    "for u, f in zip(urls, file_names):\n",
    "    path = prefix/f\n",
    "    # 不存在则直接通过 weget进行下载\n",
    "    if not path.exists():\n",
    "        !wget {u} -O {path}\n",
    "    if path.suffix == \".tgz\":\n",
    "        !tar -xvf {path} -C {prefix}\n",
    "    elif path.suffix == \".zip\":\n",
    "        !unzip -o {path} -d {prefix}\n",
    "!mv {prefix/'raw.en'} {prefix/'train_dev.raw.en'}\n",
    "!mv {prefix/'raw.zh'} {prefix/'train_dev.raw.zh'}\n",
    "!mv {prefix/'test/test.en'} {prefix/'test.raw.en'}\n",
    "!mv {prefix/'test/test.zh'} {prefix/'test.raw.zh'}\n",
    "!rm -rf {prefix/'test'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 文件数据查看"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:39:04.238203Z",
     "iopub.status.busy": "2023-03-12T11:39:04.237825Z",
     "iopub.status.idle": "2023-03-12T11:39:04.244564Z",
     "shell.execute_reply": "2023-03-12T11:39:04.243495Z",
     "shell.execute_reply.started": "2023-03-12T11:39:04.238161Z"
    }
   },
   "outputs": [],
   "source": [
    "src_lang = 'en'\n",
    "tgt_lang = 'zh'\n",
    "\n",
    "data_prefix = f'{prefix}/train_dev.raw'\n",
    "test_prefix = f'{prefix}/test.raw'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:39:04.246671Z",
     "iopub.status.busy": "2023-03-12T11:39:04.246049Z",
     "iopub.status.idle": "2023-03-12T11:39:06.174445Z",
     "shell.execute_reply": "2023-03-12T11:39:06.173225Z",
     "shell.execute_reply.started": "2023-03-12T11:39:04.246634Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thank you so much, Chris.\n",
      "And it's truly a great honor to have the opportunity to come to this stage twice; I'm extremely grateful.\n",
      "I have been blown away by this conference, and I want to thank all of you for the many nice comments about what I had to say the other night.\n",
      "And I say that sincerely, partly because  I need that.\n",
      "Put yourselves in my position.\n",
      "非常謝謝你，克里斯。能有這個機會第二度踏上這個演講台\n",
      "真是一大榮幸。我非常感激。\n",
      "這個研討會給我留下了極為深刻的印象，我想感謝大家 對我之前演講的好評。\n",
      "我是由衷的想這麼說，有部份原因是因為 —— 我真的有需要!\n",
      "請你們設身處地為我想一想！\n"
     ]
    }
   ],
   "source": [
    "# 查看数据\n",
    "!head {data_prefix+'.'+src_lang} -n 5\n",
    "!head {data_prefix+'.'+tgt_lang} -n 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据处理\n",
    "\n",
    "\n",
    "1. 全角转半角 `strQ2B`\n",
    "2. 一些字符串替换 `clean_s`\n",
    "3. 训练数据适当的清洗 `clean_corpus`\n",
    "    - 删除过短数据 `min_len`\n",
    "    - 删除过长数据 `max_len`\n",
    "    - 删除翻译前后数据长度比例超过一定值的字段 `ratio`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:39:06.176732Z",
     "iopub.status.busy": "2023-03-12T11:39:06.176354Z",
     "iopub.status.idle": "2023-03-12T11:39:06.194046Z",
     "shell.execute_reply": "2023-03-12T11:39:06.193023Z",
     "shell.execute_reply.started": "2023-03-12T11:39:06.176689Z"
    }
   },
   "outputs": [],
   "source": [
    "def strQ2B(ustring):\n",
    "    \"\"\"\n",
    "    把字符串全角转半角\n",
    "    \"\"\"\n",
    "    # 参考: https://cloud.tencent.com/developer/article/1435475\n",
    "    ss = []\n",
    "    for s in ustring:\n",
    "        rstring = \"\"\n",
    "        for uchar in s:\n",
    "            inside_code = ord(uchar)\n",
    "            if inside_code == 12288:  # 全角空格直接转换\n",
    "                inside_code = 32\n",
    "            elif (inside_code >= 65281 and inside_code <= 65374):   # 全角字符（除空格）根据关系转化\n",
    "                inside_code -= 65248\n",
    "            rstring += chr(inside_code)\n",
    "        ss.append(rstring)\n",
    "    return ''.join(ss)\n",
    "\n",
    "\n",
    "def clean_s(s, lang):\n",
    "    if lang == 'en':\n",
    "        s = re.sub(r\"\\([^()]*\\)\", \"\", s) # 删除 ([text])\n",
    "        s = s.replace('-', '') # 删除 '-'\n",
    "        s = re.sub('([.,;!?()\\\"])', r' \\1 ', s) # 保留标点符号\n",
    "    elif lang == 'zh':\n",
    "        s = strQ2B(s) # 把字符串全角转半角\n",
    "        s = re.sub(r\"\\([^()]*\\)\", \"\", s) # 删除 ([text])\n",
    "        s = s.replace(' ', '')\n",
    "        s = s.replace('—', '')\n",
    "        s = s.replace('“', '\"')\n",
    "        s = s.replace('”', '\"')\n",
    "        s = s.replace('_', '')\n",
    "        s = re.sub('([。,;!?()\\\"~「」])', r' \\1 ', s) # 保留标点符号\n",
    "    s = ' '.join(s.strip().split())\n",
    "    return s\n",
    "\n",
    "\n",
    "def len_s(s, lang):\n",
    "    if lang == 'zh':\n",
    "        return len(s)\n",
    "    return len(s.split())\n",
    "\n",
    "\n",
    "def clean_corpus(prefix, l1, l2, ratio=9, max_len=1000, min_len=1):\n",
    "    if Path(f'{prefix}.clean.{l1}').exists() and Path(f'{prefix}.clean.{l2}').exists():\n",
    "        print(f'{prefix}.clean.{l1} & {l2} exists. skipping clean.')\n",
    "        return\n",
    "    with open(f'{prefix}.{l1}', 'r') as l1_in_f:\n",
    "        with open(f'{prefix}.{l2}', 'r') as l2_in_f:\n",
    "            with open(f'{prefix}.clean.{l1}', 'w') as l1_out_f:\n",
    "                with open(f'{prefix}.clean.{l2}', 'w') as l2_out_f:\n",
    "                    for s1 in tqdm(l1_in_f):\n",
    "                        s1 = s1.strip()\n",
    "                        s2 = l2_in_f.readline().strip()\n",
    "                        s1 = clean_s(s1, l1)\n",
    "                        s2 = clean_s(s2, l2)\n",
    "                        s1_len = len_s(s1, l1)\n",
    "                        s2_len = len_s(s2, l2)\n",
    "                        if min_len > 0: # 删除过短数据\n",
    "                            if s1_len < min_len or s2_len < min_len:\n",
    "                                continue\n",
    "                        if max_len > 0: # 删除过长数据\n",
    "                            if s1_len > max_len or s2_len > max_len:\n",
    "                                continue\n",
    "                        if ratio > 0: # 删除翻译前后数据长度比例超过一定值的字段\n",
    "                            if s1_len/s2_len > ratio or s2_len/s1_len > ratio:\n",
    "                                continue\n",
    "                        print(s1, file=l1_out_f)\n",
    "                        print(s2, file=l2_out_f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:39:06.197156Z",
     "iopub.status.busy": "2023-03-12T11:39:06.195824Z",
     "iopub.status.idle": "2023-03-12T11:39:08.134093Z",
     "shell.execute_reply": "2023-03-12T11:39:08.132788Z",
     "shell.execute_reply.started": "2023-03-12T11:39:06.197114Z"
    }
   },
   "outputs": [],
   "source": [
    "!rm -f /kaggle/working/DATA/rawdata/ted2020/train_dev.raw.clean.en \n",
    "!rm -f /kaggle/working/DATA/rawdata/ted2020/train_dev.raw.clean.zh "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:39:08.138572Z",
     "iopub.status.busy": "2023-03-12T11:39:08.138185Z",
     "iopub.status.idle": "2023-03-12T11:39:24.225220Z",
     "shell.execute_reply": "2023-03-12T11:39:24.224141Z",
     "shell.execute_reply.started": "2023-03-12T11:39:08.138526Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "394066it [00:15, 24638.60it/s]\n",
      "4000it [00:00, 51828.72it/s]\n"
     ]
    }
   ],
   "source": [
    "clean_corpus(data_prefix, src_lang, tgt_lang)\n",
    "clean_corpus(test_prefix, src_lang, tgt_lang, ratio=-1, min_len=-1, max_len=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T11:39:24.227233Z",
     "iopub.status.busy": "2023-03-12T11:39:24.226865Z",
     "iopub.status.idle": "2023-03-12T11:39:26.227683Z",
     "shell.execute_reply": "2023-03-12T11:39:26.226361Z",
     "shell.execute_reply.started": "2023-03-12T11:39:24.227198Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thank you so much , Chris .\n",
      "And it's truly a great honor to have the opportunity to come to this stage twice ; I'm extremely grateful .\n",
      "I have been blown away by this conference , and I want to thank all of you for the many nice comments about what I had to say the other night .\n",
      "And I say that sincerely , partly because I need that .\n",
      "Put yourselves in my position .\n",
      "非常謝謝你 , 克里斯 。 能有這個機會第二度踏上這個演講台\n",
      "真是一大榮幸 。 我非常感激 。\n",
      "這個研討會給我留下了極為深刻的印象 , 我想感謝大家對我之前演講的好評 。\n",
      "我是由衷的想這麼說 , 有部份原因是因為我真的有需要 !\n",
      "請你們設身處地為我想一想 !\n"
     ]
    }
   ],
   "source": [
    "# view data\n",
    "!head {data_prefix+'.clean.'+src_lang} -n 5\n",
    "!head {data_prefix+'.clean.'+tgt_lang} -n 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据拆分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:05:21.235959Z",
     "iopub.status.busy": "2023-03-12T12:05:21.235444Z",
     "iopub.status.idle": "2023-03-12T12:05:21.244070Z",
     "shell.execute_reply": "2023-03-12T12:05:21.242936Z",
     "shell.execute_reply.started": "2023-03-12T12:05:21.235916Z"
    }
   },
   "outputs": [],
   "source": [
    "valid_ratio = 0.01 # 3000~4000 就足够了\n",
    "train_ratio = 1 - valid_ratio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:05:22.347422Z",
     "iopub.status.busy": "2023-03-12T12:05:22.347032Z",
     "iopub.status.idle": "2023-03-12T12:05:23.581829Z",
     "shell.execute_reply": "2023-03-12T12:05:23.580782Z",
     "shell.execute_reply.started": "2023-03-12T12:05:22.347391Z"
    }
   },
   "outputs": [],
   "source": [
    "if (prefix/f'train.clean.{src_lang}').exists() \\\n",
    "and (prefix/f'train.clean.{tgt_lang}').exists() \\\n",
    "and (prefix/f'valid.clean.{src_lang}').exists() \\\n",
    "and (prefix/f'valid.clean.{tgt_lang}').exists():\n",
    "    print(f'train/valid splits exists. skipping split.')\n",
    "else:\n",
    "    line_num = sum(1 for line in open(f'{data_prefix}.clean.{src_lang}'))\n",
    "    labels = list(range(line_num))\n",
    "    random.shuffle(labels)\n",
    "    for lang in [src_lang, tgt_lang]:\n",
    "        train_f = open(os.path.join(data_dir, dataset_name, f'train.clean.{lang}'), 'w')\n",
    "        valid_f = open(os.path.join(data_dir, dataset_name, f'valid.clean.{lang}'), 'w')\n",
    "        count = 0\n",
    "        # 基于下标拆分训练和测试集\n",
    "        for line in open(f'{data_prefix}.clean.{lang}', 'r'):\n",
    "            if labels[count]/line_num < train_ratio:\n",
    "                train_f.write(line)\n",
    "            else:\n",
    "                valid_f.write(line)\n",
    "            count += 1\n",
    "        train_f.close()\n",
    "        valid_f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-29T07:29:51.913688Z",
     "iopub.status.busy": "2023-01-29T07:29:51.912182Z",
     "iopub.status.idle": "2023-01-29T07:29:53.038154Z",
     "shell.execute_reply": "2023-01-29T07:29:53.036935Z",
     "shell.execute_reply.started": "2023-01-29T07:29:51.913615Z"
    }
   },
   "source": [
    "## &#x2728; 子字单元(`Subword Units`) \n",
    "在机器翻译中词表外词（OOV）的翻译是主要问题。 这个问题可以通过使用子字单元(`Subword Units`) 来缓解。\n",
    "- 我们使用 [sentencepiece](#kudo-richardson-2018-sentencepiece) python包\n",
    "- 选择“unigram”或“字节对编码（BPE）”算法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:05:23.756853Z",
     "iopub.status.busy": "2023-03-12T12:05:23.756069Z",
     "iopub.status.idle": "2023-03-12T12:11:14.415103Z",
     "shell.execute_reply": "2023-03-12T12:11:14.413508Z",
     "shell.execute_reply.started": "2023-03-12T12:05:23.756812Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "sentencepiece_trainer.cc(77) LOG(INFO) Starts training with : \n",
      "trainer_spec {\n",
      "  input: /kaggle/working/DATA/rawdata/ted2020/train.clean.en\n",
      "  input: /kaggle/working/DATA/rawdata/ted2020/valid.clean.en\n",
      "  input: /kaggle/working/DATA/rawdata/ted2020/train.clean.zh\n",
      "  input: /kaggle/working/DATA/rawdata/ted2020/valid.clean.zh\n",
      "  input_format: \n",
      "  model_prefix: /kaggle/working/DATA/rawdata/ted2020/spm8000\n",
      "  model_type: UNIGRAM\n",
      "  vocab_size: 8000\n",
      "  self_test_sample_size: 0\n",
      "  character_coverage: 1\n",
      "  input_sentence_size: 1000000\n",
      "  shuffle_input_sentence: 1\n",
      "  seed_sentencepiece_size: 1000000\n",
      "  shrinking_factor: 0.75\n",
      "  max_sentence_length: 4192\n",
      "  num_threads: 16\n",
      "  num_sub_iterations: 2\n",
      "  max_sentencepiece_length: 16\n",
      "  split_by_unicode_script: 1\n",
      "  split_by_number: 1\n",
      "  split_by_whitespace: 1\n",
      "  split_digits: 0\n",
      "  treat_whitespace_as_suffix: 0\n",
      "  allow_whitespace_only_pieces: 0\n",
      "  required_chars: \n",
      "  byte_fallback: 0\n",
      "  vocabulary_output_piece_score: 1\n",
      "  train_extremely_large_corpus: 0\n",
      "  hard_vocab_limit: 1\n",
      "  use_all_vocab: 0\n",
      "  unk_id: 0\n",
      "  bos_id: 1\n",
      "  eos_id: 2\n",
      "  pad_id: -1\n",
      "  unk_piece: <unk>\n",
      "  bos_piece: <s>\n",
      "  eos_piece: </s>\n",
      "  pad_piece: <pad>\n",
      "  unk_surface:  ⁇ \n",
      "  enable_differential_privacy: 0\n",
      "  differential_privacy_noise_level: 0\n",
      "  differential_privacy_clipping_threshold: 0\n",
      "}\n",
      "normalizer_spec {\n",
      "  name: nmt_nfkc_cf\n",
      "  add_dummy_prefix: 1\n",
      "  remove_extra_whitespaces: 1\n",
      "  escape_whitespaces: 1\n",
      "  normalization_rule_tsv: \n",
      "}\n",
      "denormalizer_spec {}\n",
      "trainer_interface.cc(350) LOG(INFO) SentenceIterator is not specified. Using MultiFileSentenceIterator.\n",
      "trainer_interface.cc(181) LOG(INFO) Loading corpus: /kaggle/working/DATA/rawdata/ted2020/train.clean.en\n",
      "trainer_interface.cc(181) LOG(INFO) Loading corpus: /kaggle/working/DATA/rawdata/ted2020/valid.clean.en\n",
      "trainer_interface.cc(181) LOG(INFO) Loading corpus: /kaggle/working/DATA/rawdata/ted2020/train.clean.zh\n",
      "trainer_interface.cc(181) LOG(INFO) Loading corpus: /kaggle/working/DATA/rawdata/ted2020/valid.clean.zh\n",
      "trainer_interface.cc(406) LOG(INFO) Loaded all 787960 sentences\n",
      "trainer_interface.cc(422) LOG(INFO) Adding meta_piece: <unk>\n",
      "trainer_interface.cc(422) LOG(INFO) Adding meta_piece: <s>\n",
      "trainer_interface.cc(422) LOG(INFO) Adding meta_piece: </s>\n",
      "trainer_interface.cc(427) LOG(INFO) Normalizing sentences...\n",
      "trainer_interface.cc(536) LOG(INFO) all chars count=51456919\n",
      "trainer_interface.cc(547) LOG(INFO) Done: 100% characters are covered.\n",
      "trainer_interface.cc(557) LOG(INFO) Alphabet size=5987\n",
      "trainer_interface.cc(558) LOG(INFO) Final character coverage=1\n",
      "trainer_interface.cc(590) LOG(INFO) Done! preprocessed 787960 sentences.\n",
      "unigram_model_trainer.cc(146) LOG(INFO) Making suffix array...\n",
      "unigram_model_trainer.cc(150) LOG(INFO) Extracting frequent sub strings...\n",
      "unigram_model_trainer.cc(201) LOG(INFO) Initialized 1000000 seed sentencepieces\n",
      "trainer_interface.cc(596) LOG(INFO) Tokenizing input sentences with whitespace: 787960\n",
      "trainer_interface.cc(607) LOG(INFO) Done! 844511\n",
      "unigram_model_trainer.cc(491) LOG(INFO) Using 844511 sentences for EM training\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=576502 obj=13.8106 num_tokens=3910060 num_tokens/piece=6.78239\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=503558 obj=11.7033 num_tokens=3923457 num_tokens/piece=7.79147\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=376851 obj=11.6812 num_tokens=4018358 num_tokens/piece=10.663\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=375657 obj=11.6556 num_tokens=4019831 num_tokens/piece=10.7008\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=281679 obj=11.7308 num_tokens=4170221 num_tokens/piece=14.8049\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=281604 obj=11.7052 num_tokens=4171030 num_tokens/piece=14.8117\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=211199 obj=11.8048 num_tokens=4343403 num_tokens/piece=20.5655\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=211195 obj=11.7766 num_tokens=4343817 num_tokens/piece=20.5678\n"
     ]
    }
   ],
   "source": [
    "import sentencepiece as spm\n",
    "\n",
    "vocab_size = 8000\n",
    "if (prefix/f'spm{vocab_size}.model').exists():\n",
    "    print(f'{prefix}/spm{vocab_size}.model exists. skipping spm_train.')\n",
    "else:\n",
    "    spm.SentencePieceTrainer.train(\n",
    "        input=','.join([f'{prefix}/train.clean.{src_lang}',\n",
    "                        f'{prefix}/valid.clean.{src_lang}',\n",
    "                        f'{prefix}/train.clean.{tgt_lang}',\n",
    "                        f'{prefix}/valid.clean.{tgt_lang}']),\n",
    "        model_prefix=prefix/f'spm{vocab_size}',\n",
    "        vocab_size=vocab_size,\n",
    "        character_coverage=1,\n",
    "        model_type='unigram', # 用'bpe'也行 \n",
    "        input_sentence_size=1e6,\n",
    "        shuffle_input_sentence=True,\n",
    "        normalization_rule_name='nmt_nfkc_cf',\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:11:14.419559Z",
     "iopub.status.busy": "2023-03-12T12:11:14.418281Z",
     "iopub.status.idle": "2023-03-12T12:11:37.622510Z",
     "shell.execute_reply": "2023-03-12T12:11:37.621597Z",
     "shell.execute_reply.started": "2023-03-12T12:11:14.419501Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=158394 obj=11.8925 num_tokens=4523126 num_tokens/piece=28.5562\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=158393 obj=11.8636 num_tokens=4523422 num_tokens/piece=28.5582\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=118794 obj=11.9965 num_tokens=4714089 num_tokens/piece=39.6829\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=118794 obj=11.9671 num_tokens=4714489 num_tokens/piece=39.6863\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=89095 obj=12.1203 num_tokens=4925252 num_tokens/piece=55.2809\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=89095 obj=12.0884 num_tokens=4925415 num_tokens/piece=55.2827\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=66821 obj=12.266 num_tokens=5159520 num_tokens/piece=77.214\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=66821 obj=12.2297 num_tokens=5159697 num_tokens/piece=77.2167\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=50115 obj=12.4367 num_tokens=5416808 num_tokens/piece=108.088\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=50115 obj=12.3957 num_tokens=5417088 num_tokens/piece=108.093\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=37586 obj=12.6383 num_tokens=5705549 num_tokens/piece=151.8\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=37586 obj=12.5897 num_tokens=5706003 num_tokens/piece=151.812\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=28189 obj=12.8835 num_tokens=6033473 num_tokens/piece=214.036\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=28189 obj=12.8243 num_tokens=6033406 num_tokens/piece=214.034\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=21141 obj=13.1874 num_tokens=6419108 num_tokens/piece=303.633\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=21141 obj=13.114 num_tokens=6419043 num_tokens/piece=303.63\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=15855 obj=13.5891 num_tokens=6865974 num_tokens/piece=433.048\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=15855 obj=13.4943 num_tokens=6865827 num_tokens/piece=433.039\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=11891 obj=14.1505 num_tokens=7419837 num_tokens/piece=623.988\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=11891 obj=14.0196 num_tokens=7420678 num_tokens/piece=624.058\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=8918 obj=15.0586 num_tokens=8207348 num_tokens/piece=920.313\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=8918 obj=14.84 num_tokens=8207268 num_tokens/piece=920.304\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=0 size=8800 obj=14.8902 num_tokens=8251592 num_tokens/piece=937.681\n",
      "unigram_model_trainer.cc(507) LOG(INFO) EM sub_iter=1 size=8800 obj=14.8786 num_tokens=8251621 num_tokens/piece=937.684\n",
      "trainer_interface.cc(685) LOG(INFO) Saving model: /kaggle/working/DATA/rawdata/ted2020/spm8000.model\n",
      "trainer_interface.cc(697) LOG(INFO) Saving vocabs: /kaggle/working/DATA/rawdata/ted2020/spm8000.vocab\n"
     ]
    }
   ],
   "source": [
    "# 用训练好的模型清洗数据: 将句子的起始终点加上标识：\n",
    "# 如 ▁這個 研 討 會 給我 留 下 了 極 為 深 刻 的 印 象 ▁, ▁我想 感 謝 大家 對我 之前 演講 的 好 評 \n",
    "spm_model = spm.SentencePieceProcessor(model_file=str(prefix/f'spm{vocab_size}.model'))\n",
    "in_tag = {\n",
    "    'train': 'train.clean',\n",
    "    'valid': 'valid.clean',\n",
    "    'test': 'test.raw.clean',\n",
    "}\n",
    "for split in ['train', 'valid', 'test']:\n",
    "    for lang in [src_lang, tgt_lang]:\n",
    "        out_path = prefix/f'{split}.{lang}'\n",
    "        if out_path.exists():\n",
    "            print(f\"{out_path} exists. skipping spm_encode.\")\n",
    "        else:\n",
    "            with open(prefix/f'{split}.{lang}', 'w') as out_f:\n",
    "                with open(prefix/f'{in_tag[split]}.{lang}', 'r') as in_f:\n",
    "                    for line in in_f:\n",
    "                        line = line.strip()\n",
    "                        tok = spm_model.encode(line, out_type=str)\n",
    "                        print(' '.join(tok), file=out_f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:11:37.624324Z",
     "iopub.status.busy": "2023-03-12T12:11:37.623884Z",
     "iopub.status.idle": "2023-03-12T12:11:38.680188Z",
     "shell.execute_reply": "2023-03-12T12:11:38.678944Z",
     "shell.execute_reply.started": "2023-03-12T12:11:37.624287Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "▁ 非常 謝 謝 你 ▁, ▁ 克 里 斯 ▁。 ▁ 能 有 這個 機會 第二 度 踏 上 這個 演講 台\n",
      "▁ 真 是 一 大 榮 幸 ▁。 ▁我 非常 感 激 ▁。\n",
      "▁這個 研 討 會 給我 留 下 了 極 為 深 刻 的 印 象 ▁, ▁我想 感 謝 大家 對我 之前 演講 的 好 評 ▁。\n",
      "▁我 是由 衷 的 想 這麼 說 ▁, ▁有 部份 原因 是因為 我 真的 有 需要 ▁!\n",
      "▁ 請 你們 設 身 處 地 為 我想 一 想 ▁!\n",
      "▁我 曾 搭 乘 副 總 統 專 機 八 年 ▁。\n",
      "▁現在 我 卻 必須 脫 了 鞋 子 才能 上 飛 機 ▁!\n",
      "▁讓我 跟 你們 說 一個 很 短 的故事 ▁, ▁你們 就會 明 白 我的 日 子 是 怎麼 過 的 ▁。\n",
      "▁這是 一個 真實 的故事 徹 頭 徹 尾 都是 真實 的 ▁。\n",
      "▁在我 跟 我 夫 人 蒂 佩 爾 離開 白 宮 後 我們 從 那 什 維 爾 的 家 開 車 到 東 邊 50 英 哩 外 的 一個 我們 擁有 的 小 農 場\n"
     ]
    }
   ],
   "source": [
    "!head -n 10 /kaggle/working/DATA/rawdata/ted2020/train.zh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## &#x2728; 使用fairseq将数据二进制化\n",
    "\n",
    "生成4个文件\n",
    "- train.en-zh.en.bin\n",
    "- train.en-zh.en.idx\n",
    "- train.en-zh.zh.bin\n",
    "- train.en-zh.zh.idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:11:38.689339Z",
     "iopub.status.busy": "2023-03-12T12:11:38.686837Z",
     "iopub.status.idle": "2023-03-12T12:14:41.637840Z",
     "shell.execute_reply": "2023-03-12T12:14:41.636533Z",
     "shell.execute_reply.started": "2023-03-12T12:11:38.689291Z"
    }
   },
   "outputs": [],
   "source": [
    "binpath = Path('./DATA/data-bin', dataset_name)\n",
    "if binpath.exists():\n",
    "    print(binpath, \"exists, will not overwrite!\")\n",
    "else:\n",
    "    !python -m fairseq_cli.preprocess \\\n",
    "        --source-lang {src_lang}\\\n",
    "        --target-lang {tgt_lang}\\\n",
    "        --trainpref {prefix/'train'}\\\n",
    "        --validpref {prefix/'valid'}\\\n",
    "        --testpref {prefix/'test'}\\\n",
    "        --destdir {binpath}\\\n",
    "        --joined-dictionary\\\n",
    "        --workers 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:41.640141Z",
     "iopub.status.busy": "2023-03-12T12:14:41.639731Z",
     "iopub.status.idle": "2023-03-12T12:14:43.552734Z",
     "shell.execute_reply": "2023-03-12T12:14:43.551494Z",
     "shell.execute_reply.started": "2023-03-12T12:14:41.640087Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DATA/data-bin/ted2020\n",
      "dict.en.txt\n",
      "dict.zh.txt\n",
      "preprocess.log\n",
      "test.en-zh.en.bin\n",
      "test.en-zh.en.idx\n",
      "test.en-zh.zh.bin\n",
      "test.en-zh.zh.idx\n",
      "train.en-zh.en.bin\n",
      "train.en-zh.en.idx\n",
      "train.en-zh.zh.bin\n",
      "train.en-zh.zh.idx\n",
      "valid.en-zh.en.bin\n",
      "valid.en-zh.en.idx\n",
      "valid.en-zh.zh.bin\n",
      "valid.en-zh.zh.idx\n",
      "\u0000\u0000\u0000\u0012\u0000\u0000\u0000\u0007\u0000\u0000\u0000\u0015\u0000\u0000\u0000\u0019\u0000\u0000\u0000#\u0000\u0000\u0000.\u0000\u0000\u0000\u001f\u0000\u0000\u0000\u0017\u0000\u0000\u0000\u0013\u0000\u0000\u00008\u0000\u0000\u0000\u001e",
      "\u0000\u0000\u0000<\u0000\u0000\u0000\u0016\u0000\u0000\u0000)\u0000\u0000\u0000*\u0000\u0000\u0000\"\u0000\u0000\u0000$\u0000\u0000\u0000H\u0000\u0000\u0000\u0013\u0000\u0000\u0000�\u0000\u0000\u0000=\u0000\u0000\u0000\u0016\u0000\u0000\u0000\f",
      "\u0000\u0000\u0000$\u0000\u0000\u0000\f",
      "\u0000\u0000\u0000&\u0000\u0000\u0000&\u0000\u0000\u0000\n",
      "\u0000\u0000\u0000\u0013\u0000\u0000\u0000\u000e\u0000\u0000\u0000\u0014\u0000\u0000\u0000\f",
      "\u0000\u0000\u0000#\u0000\u0000\u0000\u0017\u0000\u0000\u0000!\u0000\u0000\u0000\u000f\u0000\u0000\u0000'\u0000\u0000\u0000\u001f\u0000\u0000\u0000\u001c",
      "\u0000\u0000\u0000\u0011\u0000\u0000\u0000\u0012\u0000\u0000\u0000!\u0000\u0000\u00002\u0000\u0000\u0000\u001c",
      "\u0000\u0000\u0000\u0018\u0000\u0000\u0000\u0016\u0000\u0000\u0000\u001f\u0000\u0000\u0000B\u0000\u0000\u00007\u0000\u0000\u0000\u0007\u0000\u0000\u0000\n"
     ]
    }
   ],
   "source": [
    "print(binpath)\n",
    "!ls DATA/data-bin/ted2020 | sort\n",
    "!head -n 2 DATA/data-bin/ted2020/train.en-zh.en.idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 超参设置\n",
    "`config` 包含所有训练需要的超参数（便于后续的调参），以及模型需要存储的位置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.555500Z",
     "iopub.status.busy": "2023-03-12T12:14:43.555089Z",
     "iopub.status.idle": "2023-03-12T12:14:43.564627Z",
     "shell.execute_reply": "2023-03-12T12:14:43.563613Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.555457Z"
    }
   },
   "outputs": [],
   "source": [
    "config = Namespace(\n",
    "    datadir = \"./DATA/data-bin/ted2020\",\n",
    "    savedir = \"./checkpoints/rnn\",\n",
    "    source_lang = \"en\",\n",
    "    target_lang = \"zh\",\n",
    "    \n",
    "    # 设置cpu核数：fetching & processing data.\n",
    "    num_workers=2,  \n",
    "    # batch size 中tokens大小设置. 梯度累积增加有效批量 gradient accumulation increases the effective batchsize.\n",
    "    max_tokens=8192,\n",
    "    accum_steps=2,\n",
    "    \n",
    "    # NoamScheduler调整学习率。我们可以通过 lr_factor 调整最大lr。\n",
    "    #     NoamScheduler：可以在numpy-ml中了解详情 https://numpy-ml.readthedocs.io/en/latest/numpy_ml.neural_nets.schedulers.html\n",
    "    #         lr= lr_factor * ( model_size ** (-0.5) * min(step** (-0.5), step * warmup_steps ** (-1.5)) )\n",
    "    lr_factor=2.,\n",
    "    lr_warmup=4000,\n",
    "    \n",
    "    # 梯度裁剪norm ，防止梯度爆炸\n",
    "    clip_norm=1.0,\n",
    "    \n",
    "    # 训练最大轮次\n",
    "    max_epoch=15,\n",
    "    start_epoch=1,\n",
    "    \n",
    "    # beam search 大小\n",
    "    #    beam search 可以详细阅读《动手学深度学习》： https://d2l.ai/chapter_recurrent-modern/beam-search.html\n",
    "    beam=5, \n",
    "    # 生成序列最大长度 ax + b, x是原始序列长度\n",
    "    max_len_a=1.2, \n",
    "    max_len_b=10, \n",
    "    # 解码时，数据后处理：删除句子符号 和 jieba对句子 。\n",
    "    # when decoding, post process sentence by removing sentencepiece symbols and jieba tokenization.\n",
    "    post_process = \"sentencepiece\",\n",
    "    \n",
    "    # checkpoints\n",
    "    keep_last_epochs=5,\n",
    "    resume=None, # 如果设置，则从config.savedir 中的对应 checkpoint name 恢复\n",
    "    \n",
    "    # logging\n",
    "    use_wandb=False,\n",
    "    \n",
    "    # 随机种子seed\n",
    "    seed=73\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 日志"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.567251Z",
     "iopub.status.busy": "2023-03-12T12:14:43.566160Z",
     "iopub.status.idle": "2023-03-12T12:14:43.576827Z",
     "shell.execute_reply": "2023-03-12T12:14:43.575815Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.567207Z"
    }
   },
   "outputs": [],
   "source": [
    "logging.basicConfig(\n",
    "    format=\"%(asctime)s | %(levelname)s | %(name)s | %(message)s\",\n",
    "    datefmt=\"%Y-%m-%d %H:%M:%S\",\n",
    "    level=\"INFO\", # \"DEBUG\" \"WARNING\" \"ERROR\"\n",
    "    stream=sys.stdout,\n",
    ")\n",
    "proj = \"hw5.seq2seq\"\n",
    "logger = logging.getLogger(proj)\n",
    "if config.use_wandb:\n",
    "    import wandb\n",
    "    wandb.init(project=proj, name=Path(config.savedir).stem, config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.580378Z",
     "iopub.status.busy": "2023-03-12T12:14:43.579773Z",
     "iopub.status.idle": "2023-03-12T12:14:43.590078Z",
     "shell.execute_reply": "2023-03-12T12:14:43.589091Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.580351Z"
    }
   },
   "outputs": [],
   "source": [
    "cuda_env = utils.CudaEnvironment()\n",
    "utils.CudaEnvironment.pretty_print_cuda_env_list([cuda_env])\n",
    "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据集加载\n",
    "## 我们从fairseq借用TranslationTask\n",
    "* 用于加载上面创建的二进制数据\n",
    "* 实现数据迭代器 (dataloader)\n",
    "* 内置 task.source_dictionary 和 task.target_dictionary 同样方便\n",
    "* 实现 beach search decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.592537Z",
     "iopub.status.busy": "2023-03-12T12:14:43.591426Z",
     "iopub.status.idle": "2023-03-12T12:14:43.634465Z",
     "shell.execute_reply": "2023-03-12T12:14:43.633538Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.592502Z"
    }
   },
   "outputs": [],
   "source": [
    "from fairseq.tasks.translation import TranslationConfig, TranslationTask\n",
    "\n",
    "task_cfg = TranslationConfig(\n",
    "    data=config.datadir,\n",
    "    source_lang=config.source_lang,\n",
    "    target_lang=config.target_lang,\n",
    "    train_subset=\"train\",\n",
    "    required_seq_len_multiple=8,\n",
    "    dataset_impl=\"mmap\",\n",
    "    upsample_primary=1,\n",
    ")\n",
    "task = TranslationTask.setup_task(task_cfg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.639581Z",
     "iopub.status.busy": "2023-03-12T12:14:43.639292Z",
     "iopub.status.idle": "2023-03-12T12:14:43.681766Z",
     "shell.execute_reply": "2023-03-12T12:14:43.680799Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.639533Z"
    }
   },
   "outputs": [],
   "source": [
    "logger.info(\"loading data for epoch 1\")\n",
    "task.load_dataset(split=\"train\", epoch=1, combine=True) # combine if you have back-translation data.\n",
    "task.load_dataset(split=\"valid\", epoch=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.683947Z",
     "iopub.status.busy": "2023-03-12T12:14:43.683567Z",
     "iopub.status.idle": "2023-03-12T12:14:43.694375Z",
     "shell.execute_reply": "2023-03-12T12:14:43.693253Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.683910Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 1,\n",
      " 'source': tensor([  18,   14,    6, 2234,   60,   19,   80,    5,  256,   16,  405, 1407,\n",
      "        1706,    7,    2]),\n",
      " 'target': tensor([ 140,  690,   28,  270,   45,  151, 1142,  660,  606,  369, 3114, 2434,\n",
      "        1434,  192,    2])}\n",
      "\"Source: that's exactly what i do optical mind control .\"\n",
      "'Target: 這實在就是我所做的--光學操控思想'\n"
     ]
    }
   ],
   "source": [
    "sample = task.dataset(\"valid\")[1]\n",
    "pprint.pprint(sample)\n",
    "pprint.pprint(\n",
    "    \"Source: \" + \\\n",
    "    task.source_dictionary.string(\n",
    "        sample['source'],\n",
    "        config.post_process,\n",
    "    )\n",
    ")\n",
    "pprint.pprint(\n",
    "    \"Target: \" + \\\n",
    "    task.target_dictionary.string(\n",
    "        sample['target'],\n",
    "        config.post_process,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据生成器\n",
    "> 和 pytorch 中的 `torch.utils.data.DataLoader`类似\n",
    "\n",
    "* 控制每个`batch`包含不超过N个`token`，从而优化GPU内存效率\n",
    "* 每个`batch`对训练集进行重排序（乱序）\n",
    "* 忽略超过最大长度的句子\n",
    "* 将`batch`中的所有句子填充到相同的长度，从而实现GPU的并行计算\n",
    "* 在一个`token`中增加 `eos` 和 `shift`\n",
    "    - **训练的时候注意**: 为了训练模型以基于前缀预测下一个词语，我们将右移的目标序列作为decoder输入。\n",
    "    - 一般来说，将bos前置到目标就可以完成任务（如下所示）\n",
    "![seq2seq](https://i.imgur.com/0zeDyuI.png)\n",
    "    - 在`fairseq`中进行的处理有点不样,是将`eos`移动到最开始位置。根据经验，这也有同样的效果。示例:\n",
    "    ```\n",
    "    # output target (target) and Decoder input (prev_output_tokens): \n",
    "                   eos = 2\n",
    "                target = 419,  711,  238,  888,  792,   60,  968,    8,    2\n",
    "    prev_output_tokens = 2,  419,  711,  238,  888,  792,   60,  968,    8\n",
    "    ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.696972Z",
     "iopub.status.busy": "2023-03-12T12:14:43.696452Z",
     "iopub.status.idle": "2023-03-12T12:14:43.768066Z",
     "shell.execute_reply": "2023-03-12T12:14:43.766623Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.696938Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': tensor([723]),\n",
       " 'nsentences': 1,\n",
       " 'ntokens': 18,\n",
       " 'net_input': {'src_tokens': tensor([[   1,    1,    1,    1,    1,   18,   26,   82,    8,  480,   15,  651,\n",
       "           1361,   38,    6,  176, 2696,   39,    5,  822,   92,  260,    7,    2]]),\n",
       "  'src_lengths': tensor([19]),\n",
       "  'prev_output_tokens': tensor([[   2,  140,  296,  318, 1560,   51,  568,  316,  225, 1952,  254,   78,\n",
       "            151, 2691,    9,  215, 1680,   10,    1,    1,    1,    1,    1,    1]])},\n",
       " 'target': tensor([[ 140,  296,  318, 1560,   51,  568,  316,  225, 1952,  254,   78,  151,\n",
       "          2691,    9,  215, 1680,   10,    2,    1,    1,    1,    1,    1,    1]])}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seed=config.seed\n",
    "\n",
    "def load_data_iterator(task, split, epoch=1, max_tokens=4000, num_workers=1, cached=True):\n",
    "    batch_iterator = task.get_batch_iterator(\n",
    "        dataset=task.dataset(split),\n",
    "        max_tokens=max_tokens,\n",
    "        max_sentences=None,\n",
    "        # fairseq.utils\n",
    "        max_positions=utils.resolve_max_positions(\n",
    "            task.max_positions(),\n",
    "            max_tokens,\n",
    "        ),\n",
    "        ignore_invalid_inputs=True,\n",
    "        seed=seed,\n",
    "        num_workers=num_workers,\n",
    "        epoch=epoch,\n",
    "        disable_iterator_cache=not cached,\n",
    "        # 将此设置为False以加快速度。但是，如果设置为False，则将max_tokens更改为\n",
    "        # 此方法的第一次调用无效. \n",
    "    )\n",
    "    return batch_iterator\n",
    "\n",
    "demo_epoch_obj = load_data_iterator(task, \"valid\", epoch=1, max_tokens=20, num_workers=1, cached=False)\n",
    "demo_iter = demo_epoch_obj.next_epoch_itr(shuffle=True)\n",
    "sample = next(demo_iter)\n",
    "sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 每个batch是python dict, key是`string`, values是`Tensor value`.\n",
    "```python\n",
    "batch = {\n",
    "    \"id\": id, # 样例的id\n",
    "    \"nsentences\": len(samples), # 句子批次大小 (sentences)\n",
    "    \"ntokens\": ntokens, # token批次大小 (tokens)\n",
    "    \"net_input\": {\n",
    "        \"src_tokens\": src_tokens, # 翻译的句子(`source language`)\n",
    "        \"src_lengths\": src_lengths, # 句子padding前的长度\n",
    "        \"prev_output_tokens\": prev_output_tokens, # 右移动的目标（`right shifted target`)\n",
    "    },\n",
    "    \"target\": target, # 翻译结果\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#   &#x2728; 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.772366Z",
     "iopub.status.busy": "2023-03-12T12:14:43.772034Z",
     "iopub.status.idle": "2023-03-12T12:14:43.779641Z",
     "shell.execute_reply": "2023-03-12T12:14:43.778734Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.772327Z"
    }
   },
   "outputs": [],
   "source": [
    "from fairseq.models import (\n",
    "    FairseqEncoder, \n",
    "    FairseqIncrementalDecoder,\n",
    "    FairseqEncoderDecoderModel\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The Encoder 可以是 `RNN`  或 `Transformer Encoder`. 下面的描述是针对RNN的. 对于每个输入 token, Encoder 会生成一个 output vector 和一个 hidden states vector, 并且 hidden states vector 将传入下一个rnn cell. 换句话来说, Encoder 按输入顺序顺序读取句子, 并在每个时间步输出一个 single vector, 最终输出final hidden states, 或者 content vector, 在最后一个时间步.\n",
    "- 超参数:\n",
    "  - *args*\n",
    "      - encoder_embed_dim: embeddings维度, 将 one-hot 向量压缩, 实现维度降低\n",
    "      - encoder_ffn_embed_dim：  hidden states 和 output vectors 的维度\n",
    "      - encoder_layers：RNN Encoder的层数\n",
    "      - dropout：将部分神经元激活的概率被设置为0，以防止过度拟合。通常，这在训练中应用，在测试中删除。\n",
    "  - *dictionary*: fairseq 中提供的dictionary.它用于获得填充索引(`padding index`)，进而获得编码器填充掩码(`encoder padding mask`)。\n",
    "  - *embed_tokens*: an instance of token embeddings (nn.Embedding)\n",
    "\n",
    "- 输入: \n",
    "    - *src_tokens*: 表示英语的整数序列 e.g. 1, 28, 29, 205, 2 \n",
    "- 输出: \n",
    "    - *outputs*:  RNN每步输出,可以进行Attention处理\n",
    "    - *final_hiddens*: 每一步的`hidden states`, 传入解码器(`decoder`)用于翻译\n",
    "    - *encoder_padding_mask*: 这告诉解码器(`decoder`)忽略哪个位置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.783906Z",
     "iopub.status.busy": "2023-03-12T12:14:43.783526Z",
     "iopub.status.idle": "2023-03-12T12:14:43.799394Z",
     "shell.execute_reply": "2023-03-12T12:14:43.798153Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.783876Z"
    }
   },
   "outputs": [],
   "source": [
    "class RNNEncoder(FairseqEncoder):\n",
    "    def __init__(self, args, dictionary, embed_tokens):\n",
    "        super().__init__(dictionary)\n",
    "        self.embed_tokens = embed_tokens\n",
    "        \n",
    "        self.embed_dim = args.encoder_embed_dim\n",
    "        self.hidden_dim = args.encoder_ffn_embed_dim\n",
    "        self.num_layers = args.encoder_layers\n",
    "        \n",
    "        self.dropout_in_module = nn.Dropout(args.dropout)\n",
    "        self.rnn = nn.GRU(\n",
    "            self.embed_dim, \n",
    "            self.hidden_dim, \n",
    "            self.num_layers, \n",
    "            dropout=args.dropout, \n",
    "            batch_first=False, \n",
    "            bidirectional=True\n",
    "        )\n",
    "        self.dropout_out_module = nn.Dropout(args.dropout)\n",
    "        \n",
    "        self.padding_idx = dictionary.pad()\n",
    "        \n",
    "    def combine_bidir(self, outs, bsz: int):\n",
    "        out = outs.view(self.num_layers, 2, bsz, -1).transpose(1, 2).contiguous()\n",
    "        return out.view(self.num_layers, bsz, -1)\n",
    "\n",
    "    def forward(self, src_tokens, **unused):\n",
    "        bsz, seqlen = src_tokens.size()\n",
    "        \n",
    "        # 获取 embeddings\n",
    "        x = self.embed_tokens(src_tokens)\n",
    "        x = self.dropout_in_module(x)\n",
    "\n",
    "        # B x T x C -> T x B x C\n",
    "        x = x.transpose(0, 1)\n",
    "        \n",
    "        # 直通双向 RNN\n",
    "        h0 = x.new_zeros(2 * self.num_layers, bsz, self.hidden_dim)\n",
    "        x, final_hiddens = self.rnn(x, h0)\n",
    "        outputs = self.dropout_out_module(x)\n",
    "        # outputs = [sequence len, batch size, hid dim * directions]\n",
    "        # hidden =  [num_layers * directions, batch size  , hid dim] \n",
    "        \n",
    "        # 由于encode是双向的，我们需要连接两个方向的隐藏状态\n",
    "        final_hiddens = self.combine_bidir(final_hiddens, bsz)\n",
    "        # hidden =  [num_layers x batch x num_directions*hidden]\n",
    "        \n",
    "        encoder_padding_mask = src_tokens.eq(self.padding_idx).t()\n",
    "        return tuple(\n",
    "            (\n",
    "                outputs,  # seq_len x batch x hidden\n",
    "                final_hiddens,  # num_layers x batch x num_directions*hidden\n",
    "                encoder_padding_mask,  # seq_len x batch\n",
    "            )\n",
    "        )\n",
    "    \n",
    "    def reorder_encoder_out(self, encoder_out, new_order):\n",
    "        # 这部分会在fairseq's beam search中使用。\n",
    "        return tuple(\n",
    "            (\n",
    "                encoder_out[0].index_select(1, new_order),\n",
    "                encoder_out[1].index_select(1, new_order),\n",
    "                encoder_out[2].index_select(1, new_order),\n",
    "            )\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention\n",
    "- 当输入序列较长时，“内容向量”单独不能准确地表示整个序列，注意力机制(`Attention`)可以为解码器(`Encoder`)提供更多信息。\n",
    "- 根据当前时间步长的**Decoder embeddings**，将**Encoder outputs**与解码器嵌入(`Decoder embeddings`)匹配，以确定相关性，然后将编码器输出与相关性加权相加，作为**Decoder**RNN的输入。 \n",
    "- 常见的注意力实现使用神经网络/点积作为**query**（解码器嵌入`decoder embeddings`）和**key**（编码器输出`Encoder outputs`）之间的相关性，然后是**softmax**以获得分布，最后**values**（编码器输入`Encoder outputs`）被所述分布**加权和**。\n",
    "    - $Q = W_Q  I_{decoder-emb}$\n",
    "    - $K = W_K  I_{encoder-out}$\n",
    "    - $V = W_V  I_{encoder-out}$\n",
    "    - $A = K^TQ$\n",
    "    - $A'= softmax(A)$\n",
    "    - $O = VA'$\n",
    "\n",
    "- Parameters:\n",
    "  - *source_embed_dim*: query的维度 $W_Q$\n",
    "  - *input_embed_dim*: key的维度 $W_K$\n",
    "  - *output_embed_dim*: value的维度 $W_V$\n",
    "\n",
    "- Inputs: \n",
    "    - *inputs*: 做attention的输入\n",
    "    - *encoder_outputs*:  作为 query / value,\n",
    "    - *encoder_padding_mask*: 这告诉解码器`decoder`忽略哪个位置\n",
    "- Outputs: \n",
    "    - *output*: attention后的上下文向量\n",
    "    - *attention score*: attention后的分布 $A'$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.803668Z",
     "iopub.status.busy": "2023-03-12T12:14:43.803175Z",
     "iopub.status.idle": "2023-03-12T12:14:43.817566Z",
     "shell.execute_reply": "2023-03-12T12:14:43.816420Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.803640Z"
    }
   },
   "outputs": [],
   "source": [
    "class AttentionLayer(nn.Module):\n",
    "    def __init__(self, input_embed_dim, source_embed_dim, output_embed_dim, bias=False):\n",
    "        \"\"\"\n",
    "        计算decoder embeding 之后 和 encoder out之间的相关程度\n",
    "        params:\n",
    "            source_embed_dim: query的维度 $W_Q * I_{decoder-emb}$\n",
    "            input_embed_dim: key的维度 $W_K * I_{encoder-out}$\n",
    "            output_embed_dim: value的维度 $W_V * I_{encoder-out}$\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "\n",
    "        self.Q = nn.Linear(input_embed_dim, source_embed_dim, bias=bias)\n",
    "        self.output_proj = nn.Linear(\n",
    "            input_embed_dim + source_embed_dim, output_embed_dim, bias=bias\n",
    "        )\n",
    "    \n",
    "    # 这里对encoder-out 不做linear变换\n",
    "    def K(self, input_tensor):\n",
    "        return input_tensor\n",
    "    \n",
    "    # 这里对encoder-out 不做linear变换\n",
    "    def V(self, input_tensor):\n",
    "        return input_tensor\n",
    "\n",
    "    def forward(self, inputs, encoder_outputs, encoder_padding_mask):\n",
    "        # inputs: T, B, dim\n",
    "        # encoder_outputs: S x B x dim\n",
    "        # padding mask:  S x B\n",
    "        \n",
    "        # 将Batch的维度放在第一\n",
    "        inputs = inputs.transpose(1,0) # B, T, dim\n",
    "        encoder_outputs = encoder_outputs.transpose(1,0) # B, S, dim\n",
    "        encoder_padding_mask = encoder_padding_mask.transpose(1,0) # B, S\n",
    "        \n",
    "        # Q = W_QI_{decode-emb} 投影到编码器输出的维度\n",
    "        x = self.Q(inputs)\n",
    "\n",
    "        # 计算 attention\n",
    "        # (B, T, dim) x (B, dim, S) = (B, T, S)\n",
    "        # A = K^TQ\n",
    "        attn_scores = torch.bmm(x, self.K(encoder_outputs.transpose(1,2)))\n",
    "        # 在与padding相对应的位置取消注意\n",
    "        if encoder_padding_mask is not None:\n",
    "            # B, S -> (B, 1, S)\n",
    "            encoder_padding_mask = encoder_padding_mask.unsqueeze(1)\n",
    "            attn_scores = (\n",
    "                attn_scores.float()\n",
    "                .masked_fill_(encoder_padding_mask, float(\"-inf\"))\n",
    "                .type_as(attn_scores)\n",
    "            )  # FP16 support: cast to float and back\n",
    "\n",
    "        # A' = softmax(A)\n",
    "        attn_scores = F.softmax(attn_scores, dim=-1)\n",
    "\n",
    "        # O = V A'   (B, T, S) x (B, S, dim) = (B, T, dim) 加权和\n",
    "        x = torch.bmm(attn_scores, self.V(encoder_outputs))\n",
    "\n",
    "        # 最终合并 I 和 O  (B, T, dim)\n",
    "        x = torch.cat((x, inputs), dim=-1)\n",
    "        x = torch.tanh(self.output_proj(x)) # concat + linear + tanh\n",
    "        \n",
    "        # (B, T, dim) -> (T, B, dim)\n",
    "        return x.transpose(1,0), attn_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder\n",
    "\n",
    "* **Decoder**的隐藏状态(`hidden states`)将由**Encoder**的最终隐藏状态（`final hidden states`）初始化\n",
    "* 同时，**Decoder**将根据当前时间步长的输入（之前时间步长的输出）更改其隐藏状态（`hidden states`），并生成输出\n",
    "* Attention提高了性能\n",
    "* seq2seq步骤在解码器(`decoder`)中实现 , 以便以后Seq2Seq类可以接受RNN和Transformer，而无需进一步修改。\n",
    "\n",
    "- Parameters:\n",
    "  - *args*\n",
    "      - decoder_embed_dim: decoder 维度, 和`encoder_embed_dim`类似，\n",
    "      - decoder_ffn_embed_dim: decoder RNN 的隐含层(hidden states`)维度,和`encoder_ffn_embed_dim`类似\n",
    "      - decoder_layers: decoder RNN 的网络层数\n",
    "      - share_decoder_input_output_embed: 通常，解码器`decoder`的投影矩阵将与解码器输入嵌入(`decoder input embeddings`)共享权重\n",
    "  - *dictionary*: fairseq 中提供的dictionary.\n",
    "  - *embed_tokens*: an instance of token embeddings (nn.Embedding)\n",
    "\n",
    "- Inputs: \n",
    "    - *prev_output_tokens*: 表示右移目标(`right-shifted target`)的整数序列  e.g. 1, 28, 29, 205, 2 \n",
    "    - *encoder_out*: encoder的输出.\n",
    "    - *incremental_state*: 为了在测试期间加快解码速度，我们将保存每个时间步长的隐藏状态(` hidden state`)。\n",
    "\n",
    "- Outputs: \n",
    "    - *outputs*: 解码器每个时间步的logits（softmax之前）输出\n",
    "    - *extra*: 没有使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.820176Z",
     "iopub.status.busy": "2023-03-12T12:14:43.819722Z",
     "iopub.status.idle": "2023-03-12T12:14:43.841055Z",
     "shell.execute_reply": "2023-03-12T12:14:43.839965Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.820116Z"
    }
   },
   "outputs": [],
   "source": [
    "class RNNDecoder(FairseqIncrementalDecoder):\n",
    "    def __init__(self, args, dictionary, embed_tokens):\n",
    "        super().__init__(dictionary)\n",
    "        self.embed_tokens = embed_tokens\n",
    "        \n",
    "        assert args.decoder_layers == args.encoder_layers, f\"\"\"seq2seq rnn requires that encoder \n",
    "        and decoder have same layers of rnn. got: {args.encoder_layers, args.decoder_layers}\"\"\"\n",
    "        assert args.decoder_ffn_embed_dim == args.encoder_ffn_embed_dim*2, f\"\"\"seq2seq-rnn requires \n",
    "        that decoder hidden to be 2*encoder hidden dim. got: {args.decoder_ffn_embed_dim, args.encoder_ffn_embed_dim*2}\"\"\"\n",
    "        \n",
    "        self.embed_dim = args.decoder_embed_dim\n",
    "        self.hidden_dim = args.decoder_ffn_embed_dim\n",
    "        self.num_layers = args.decoder_layers\n",
    "        \n",
    "        \n",
    "        self.dropout_in_module = nn.Dropout(args.dropout)\n",
    "        self.rnn = nn.GRU(\n",
    "            self.embed_dim, \n",
    "            self.hidden_dim, \n",
    "            self.num_layers, \n",
    "            dropout=args.dropout, \n",
    "            batch_first=False, \n",
    "            bidirectional=False\n",
    "        )\n",
    "        self.attention = AttentionLayer(\n",
    "            self.embed_dim, self.hidden_dim, self.embed_dim, bias=False\n",
    "        ) \n",
    "        # self.attention = None\n",
    "        self.dropout_out_module = nn.Dropout(args.dropout)\n",
    "        \n",
    "        if self.hidden_dim != self.embed_dim:\n",
    "            self.project_out_dim = nn.Linear(self.hidden_dim, self.embed_dim)\n",
    "        else:\n",
    "            self.project_out_dim = None\n",
    "        \n",
    "        if args.share_decoder_input_output_embed:\n",
    "            self.output_projection = nn.Linear(\n",
    "                self.embed_tokens.weight.shape[1],\n",
    "                self.embed_tokens.weight.shape[0],\n",
    "                bias=False,\n",
    "            )\n",
    "            self.output_projection.weight = self.embed_tokens.weight\n",
    "        else:\n",
    "            self.output_projection = nn.Linear(\n",
    "                self.output_embed_dim, len(dictionary), bias=False\n",
    "            )\n",
    "            nn.init.normal_(\n",
    "                self.output_projection.weight, mean=0, std=self.output_embed_dim ** -0.5\n",
    "            )\n",
    "        \n",
    "    def forward(self, prev_output_tokens, encoder_out, incremental_state=None, **unused):\n",
    "        # 从编码器encoder中提取输出\n",
    "        encoder_outputs, encoder_hiddens, encoder_padding_mask = encoder_out\n",
    "        # outputs:          seq_len x batch x num_directions*hidden\n",
    "        # encoder_hiddens:  num_layers x batch x num_directions*encoder_hidden\n",
    "        # padding_mask:     seq_len x batch\n",
    "        \n",
    "        if incremental_state is not None and len(incremental_state) > 0:\n",
    "            # 如果保留了上一个时间步的信息，我们可以从那里继续，而不是从头开始\n",
    "            prev_output_tokens = prev_output_tokens[:, -1:]\n",
    "            cache_state = self.get_incremental_state(incremental_state, \"cached_state\")\n",
    "            prev_hiddens = cache_state[\"prev_hiddens\"]\n",
    "        else:\n",
    "            # 增量状态不存在，或者这是训练时间，或者是测试时间的第一个时间步\n",
    "            # 准备 seq2seq: 将encoder_hidden传递给解码器decoder隐藏状态 \n",
    "            prev_hiddens = encoder_hiddens\n",
    "        \n",
    "        bsz, seqlen = prev_output_tokens.size()\n",
    "        \n",
    "        # embed tokens\n",
    "        x = self.embed_tokens(prev_output_tokens)\n",
    "        x = self.dropout_in_module(x)\n",
    "\n",
    "        # B x T x C -> T x B x C\n",
    "        x = x.transpose(0, 1)\n",
    "                \n",
    "        # decoder-to-encoder attention\n",
    "        if self.attention is not None:\n",
    "            x, attn = self.attention(x, encoder_outputs, encoder_padding_mask)\n",
    "                        \n",
    "        # 直通双向 RNN\n",
    "        x, final_hiddens = self.rnn(x, prev_hiddens)\n",
    "        # outputs = [sequence len, batch size, hid dim]\n",
    "        # hidden =  [num_layers * directions, batch size  , hid dim]\n",
    "        x = self.dropout_out_module(x)\n",
    "                \n",
    "        # 投影到 embedding size （如果hidden与嵌入embed大小不同，且share_embedding为True，则需要进行额外的投影操作）\n",
    "        if self.project_out_dim != None:\n",
    "            x = self.project_out_dim(x)\n",
    "        \n",
    "        # 投影到词表大小 vocab size\n",
    "        x = self.output_projection(x)\n",
    "        \n",
    "        # T x B x C -> B x T x C\n",
    "        x = x.transpose(1, 0)\n",
    "        \n",
    "        # 如果是增量，记录当前时间步的隐藏状态，将在下一个时间步中恢复\n",
    "        cache_state = {\n",
    "            \"prev_hiddens\": final_hiddens,\n",
    "        }\n",
    "        self.set_incremental_state(incremental_state, \"cached_state\", cache_state)\n",
    "        \n",
    "        return x, None\n",
    "    \n",
    "    def reorder_incremental_state(\n",
    "        self,\n",
    "        incremental_state,\n",
    "        new_order,\n",
    "    ):\n",
    "        # 在fairseq's beam search中使用\n",
    "        cache_state = self.get_incremental_state(incremental_state, \"cached_state\")\n",
    "        prev_hiddens = cache_state[\"prev_hiddens\"]\n",
    "        prev_hiddens = [p.index_select(0, new_order) for p in prev_hiddens]\n",
    "        cache_state = {\n",
    "            \"prev_hiddens\": torch.stack(prev_hiddens),\n",
    "        }\n",
    "        self.set_incremental_state(incremental_state, \"cached_state\", cache_state)\n",
    "        return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-29T08:55:23.732988Z",
     "iopub.status.busy": "2023-01-29T08:55:23.732618Z",
     "iopub.status.idle": "2023-01-29T08:55:23.738442Z",
     "shell.execute_reply": "2023-01-29T08:55:23.736857Z",
     "shell.execute_reply.started": "2023-01-29T08:55:23.732957Z"
    }
   },
   "source": [
    "## Seq2seq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.843594Z",
     "iopub.status.busy": "2023-03-12T12:14:43.842813Z",
     "iopub.status.idle": "2023-03-12T12:14:43.852086Z",
     "shell.execute_reply": "2023-03-12T12:14:43.851094Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.843557Z"
    }
   },
   "outputs": [],
   "source": [
    "class Seq2Seq(FairseqEncoderDecoderModel):\n",
    "    def __init__(self, args, encoder, decoder):\n",
    "        super().__init__(encoder, decoder)\n",
    "        self.args = args\n",
    "    \n",
    "    def forward(\n",
    "        self,\n",
    "        src_tokens,\n",
    "        src_lengths,\n",
    "        prev_output_tokens,\n",
    "        return_all_hiddens: bool = True,\n",
    "    ):\n",
    "        \"\"\"\n",
    "        前向传播： encoder -> decoder\n",
    "        \"\"\"\n",
    "        encoder_out = self.encoder(\n",
    "            src_tokens, src_lengths=src_lengths, return_all_hiddens=return_all_hiddens\n",
    "        )\n",
    "        logits, extra = self.decoder(\n",
    "            prev_output_tokens,\n",
    "            encoder_out=encoder_out,\n",
    "            src_lengths=src_lengths,\n",
    "            return_all_hiddens=return_all_hiddens,\n",
    "        )\n",
    "        return logits, extra"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## &#x2728; 模型初始化\n",
    "\n",
    "<font color=darkred><b>***TODO***: Encoder和Decoder修改：改用 TransformerEncoder & TransformerDecoder</font></b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.854206Z",
     "iopub.status.busy": "2023-03-12T12:14:43.853822Z",
     "iopub.status.idle": "2023-03-12T12:14:43.867728Z",
     "shell.execute_reply": "2023-03-12T12:14:43.866900Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.854161Z"
    }
   },
   "outputs": [],
   "source": [
    "# 提示: transformer 结构\n",
    "from fairseq.models.transformer import (\n",
    "    TransformerEncoder, \n",
    "    TransformerDecoder,\n",
    ")\n",
    "\n",
    "def build_model(args, task):\n",
    "    \"\"\"基于超参数构建模型实例\"\"\"\n",
    "    src_dict, tgt_dict = task.source_dictionary, task.target_dictionary\n",
    "\n",
    "    # token embeddings\n",
    "    encoder_embed_tokens = nn.Embedding(len(src_dict), args.encoder_embed_dim, src_dict.pad())\n",
    "    decoder_embed_tokens = nn.Embedding(len(tgt_dict), args.decoder_embed_dim, tgt_dict.pad())\n",
    "    \n",
    "    # encoder decoder\n",
    "    # 提示: TODO: 改用 TransformerEncoder & TransformerDecoder\n",
    "    encoder = RNNEncoder(args, src_dict, encoder_embed_tokens)\n",
    "    decoder = RNNDecoder(args, tgt_dict, decoder_embed_tokens)\n",
    "    # encoder = TransformerEncoder(args, src_dict, encoder_embed_tokens)\n",
    "    # decoder = TransformerDecoder(args, tgt_dict, decoder_embed_tokens)\n",
    "\n",
    "    # sequence to sequence model\n",
    "    model = Seq2Seq(args, encoder, decoder)\n",
    "    \n",
    "    # seq2seq 模型初始化很重要, 参数权重的初始化需要一些其他操作\n",
    "    def init_params(module):\n",
    "        from fairseq.modules import MultiheadAttention\n",
    "        if isinstance(module, nn.Linear):\n",
    "            module.weight.data.normal_(mean=0.0, std=0.02)\n",
    "            if module.bias is not None:\n",
    "                module.bias.data.zero_()\n",
    "        if isinstance(module, nn.Embedding):\n",
    "            module.weight.data.normal_(mean=0.0, std=0.02)\n",
    "            if module.padding_idx is not None:\n",
    "                module.weight.data[module.padding_idx].zero_()\n",
    "        if isinstance(module, MultiheadAttention):\n",
    "            module.q_proj.weight.data.normal_(mean=0.0, std=0.02)\n",
    "            module.k_proj.weight.data.normal_(mean=0.0, std=0.02)\n",
    "            module.v_proj.weight.data.normal_(mean=0.0, std=0.02)\n",
    "        if isinstance(module, nn.RNNBase):\n",
    "            for name, param in module.named_parameters():\n",
    "                if \"weight\" in name or \"bias\" in name:\n",
    "                    param.data.uniform_(-0.1, 0.1)\n",
    "    # 权重初始化\n",
    "    model.apply(init_params)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型框架一些参数配置\n",
    "\n",
    "对于强基线，请参考表3中*transformer-base*的超参数 [Attention is all you need](#vaswani2017)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.869895Z",
     "iopub.status.busy": "2023-03-12T12:14:43.869268Z",
     "iopub.status.idle": "2023-03-12T12:14:43.880665Z",
     "shell.execute_reply": "2023-03-12T12:14:43.879675Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.869860Z"
    }
   },
   "outputs": [],
   "source": [
    "arch_args = Namespace(\n",
    "    encoder_embed_dim=256,\n",
    "    encoder_ffn_embed_dim=512,\n",
    "    encoder_layers=1,\n",
    "    decoder_embed_dim=256,\n",
    "    decoder_ffn_embed_dim=1024,\n",
    "    decoder_layers=1,\n",
    "    share_decoder_input_output_embed=True,\n",
    "    dropout=0.3,\n",
    ")\n",
    "\n",
    "# 提示: 这些是关于Transformer参数的补丁 \n",
    "def add_transformer_args(args):\n",
    "    args.encoder_attention_heads=4\n",
    "    args.encoder_normalize_before=True\n",
    "    \n",
    "    args.decoder_attention_heads=4\n",
    "    args.decoder_normalize_before=True\n",
    "    \n",
    "    args.activation_fn=\"relu\"\n",
    "    args.max_source_positions=1024\n",
    "    args.max_target_positions=1024\n",
    "    \n",
    "    # Transformer默认参数上的修补程序 (以上未列出)\n",
    "    from fairseq.models.transformer import base_architecture\n",
    "    base_architecture(arch_args)\n",
    "\n",
    "# add_transformer_args(arch_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:43.882587Z",
     "iopub.status.busy": "2023-03-12T12:14:43.882201Z",
     "iopub.status.idle": "2023-03-12T12:14:44.131401Z",
     "shell.execute_reply": "2023-03-12T12:14:44.130369Z",
     "shell.execute_reply.started": "2023-03-12T12:14:43.882534Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/rnn.py:65: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.3 and num_layers=1\n",
      "  \"num_layers={}\".format(dropout, num_layers))\n"
     ]
    }
   ],
   "source": [
    "if config.use_wandb:\n",
    "    wandb.config.update(vars(arch_args))\n",
    "\n",
    "    \n",
    "model = build_model(arch_args, task)\n",
    "logger.info(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 优化器\n",
    "\n",
    "## 损失函数: 标签平滑正则化（`Label Smoothing Regularization`)\n",
    "* 让模型学习生成不太集中的分布，并防止过度自信(`over-confidence`)\n",
    "* 有时，事实真相可能不是唯一的答案。因此，在计算损失时，我们为不正确的标签保留一些概率\n",
    "* 防止过拟合\n",
    "\n",
    "code [来源：fairseq/criterions/label_smoothed_cross_entropy](https://fairseq.readthedocs.io/en/latest/_modules/fairseq/criterions/label_smoothed_cross_entropy.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:14:44.133849Z",
     "iopub.status.busy": "2023-03-12T12:14:44.132912Z",
     "iopub.status.idle": "2023-03-12T12:14:44.156415Z",
     "shell.execute_reply": "2023-03-12T12:14:44.155357Z",
     "shell.execute_reply.started": "2023-03-12T12:14:44.133809Z"
    }
   },
   "outputs": [],
   "source": [
    "class LabelSmoothedCrossEntropyCriterion(nn.Module):\n",
    "    def __init__(self, smoothing, ignore_index=None, reduce=True):\n",
    "        super().__init__()\n",
    "        self.smoothing = smoothing\n",
    "        self.ignore_index = ignore_index\n",
    "        self.reduce = reduce\n",
    "    \n",
    "    def forward(self, lprobs, target):\n",
    "        if target.dim() == lprobs.dim() - 1:\n",
    "            target = target.unsqueeze(-1)\n",
    "        # nll: 负对数似然（Negative log likelihood），当目标是一个one-ho时的交叉熵。以下行与F.nll_loss相同\n",
    "        nll_loss = -lprobs.gather(dim=-1, index=target)\n",
    "        #  为其他标签保留一些可能性。因此当计算交叉熵时，相当于对所有标签的对数概率求和\n",
    "        smooth_loss = -lprobs.sum(dim=-1, keepdim=True)\n",
    "        if self.ignore_index is not None:\n",
    "            pad_mask = target.eq(self.ignore_index)\n",
    "            nll_loss.masked_fill_(pad_mask, 0.0)\n",
    "            smooth_loss.masked_fill_(pad_mask, 0.0)\n",
    "        else:\n",
    "            nll_loss = nll_loss.squeeze(-1)\n",
    "            smooth_loss = smooth_loss.squeeze(-1)\n",
    "        if self.reduce:\n",
    "            nll_loss = nll_loss.sum()\n",
    "            smooth_loss = smooth_loss.sum()\n",
    "        # 在计算交叉熵时，添加其他标签的损失\n",
    "        eps_i = self.smoothing / lprobs.size(-1)\n",
    "        loss = (1.0 - self.smoothing) * nll_loss + eps_i * smooth_loss\n",
    "        return loss\n",
    "\n",
    "# 通常来说, 0.1 就足够了\n",
    "criterion = LabelSmoothedCrossEntropyCriterion(\n",
    "    smoothing=0.1,\n",
    "    ignore_index=task.target_dictionary.pad(),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## &#x2728; 优化器: Adam + lr scheduling\n",
    "在训练Transformer时，逆平方根学习率变化对稳定性很重要. 它后来也用于RNN。\n",
    "根据以下公式更新学习率`learning rate` . 在第一阶段，线性增加学习率，然后学习率与时间步长的平方根倒数成比例衰减。\n",
    "$$lrate = d_{\\text{model}}^{-0.5}\\cdot\\min({step\\_num}^{-0.5},{step\\_num}\\cdot{warmup\\_steps}^{-1.5})$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:23:21.125302Z",
     "iopub.status.busy": "2023-03-12T12:23:21.124872Z",
     "iopub.status.idle": "2023-03-12T12:23:21.131673Z",
     "shell.execute_reply": "2023-03-12T12:23:21.130334Z",
     "shell.execute_reply.started": "2023-03-12T12:23:21.125266Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_rate(d_model, step_num, warmup_step):\n",
    "    # TODO: 基于上述公式更新学习率\n",
    "    # lr = 0.001\n",
    "    lr = np.power(d_model, -0.5) * min(np.power(step_num, -0.5), step_num * np.power(warmup_step, -1.5))\n",
    "    return lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:23:21.705795Z",
     "iopub.status.busy": "2023-03-12T12:23:21.704701Z",
     "iopub.status.idle": "2023-03-12T12:23:21.715130Z",
     "shell.execute_reply": "2023-03-12T12:23:21.714041Z",
     "shell.execute_reply.started": "2023-03-12T12:23:21.705742Z"
    }
   },
   "outputs": [],
   "source": [
    "# 可以看：https://nn.labml.ai/optimizers/noam.html\n",
    "class NoamOpt:\n",
    "    \"实现速率的Optim包装器.\"\n",
    "    def __init__(self, model_size, factor, warmup, optimizer):\n",
    "        self.optimizer = optimizer\n",
    "        self._step = 0\n",
    "        self.warmup = warmup\n",
    "        self.factor = factor\n",
    "        self.model_size = model_size\n",
    "        self._rate = 0\n",
    "    \n",
    "    @property\n",
    "    def param_groups(self):\n",
    "        return self.optimizer.param_groups\n",
    "        \n",
    "    def multiply_grads(self, c):\n",
    "        \"\"\"将梯度乘以常数*c*.\"\"\"                \n",
    "        for group in self.param_groups:\n",
    "            for p in group['params']:\n",
    "                if p.grad is not None:\n",
    "                    p.grad.data.mul_(c)\n",
    "        \n",
    "    def step(self):\n",
    "        \"更新 parameters 和 rate\"\n",
    "        self._step += 1\n",
    "        rate = self.rate()\n",
    "        for p in self.param_groups:\n",
    "            p['lr'] = rate\n",
    "        self._rate = rate\n",
    "        self.optimizer.step()\n",
    "        \n",
    "    def rate(self, step = None):\n",
    "        \"实现上面的lrate\"\n",
    "        if step is None:\n",
    "            step = self._step\n",
    "        return 0 if not step else self.factor * get_rate(self.model_size, step, self.warmup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-29T08:55:29.371753Z",
     "iopub.status.busy": "2023-01-29T08:55:29.370347Z",
     "iopub.status.idle": "2023-01-29T08:55:29.376971Z",
     "shell.execute_reply": "2023-01-29T08:55:29.375625Z",
     "shell.execute_reply.started": "2023-01-29T08:55:29.371707Z"
    }
   },
   "source": [
    "## 学习率Scheduling图示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:25:39.950346Z",
     "iopub.status.busy": "2023-03-12T12:25:39.949988Z",
     "iopub.status.idle": "2023-03-12T12:25:41.126170Z",
     "shell.execute_reply": "2023-03-12T12:25:41.125200Z",
     "shell.execute_reply.started": "2023-03-12T12:25:39.950315Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa0AAAEcCAYAAAB0wOvnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABCn0lEQVR4nO3deXyU1b348c83O4RshACRsAooARQxsvS6tGoF1IrValEr2motbb321v5uq731dtXa5fa2ttSltlZalXprVapQamldUBFREFlEwyKENWyZANmGfH9/PGfCMEwyT5JJJpl836/XvGbmeZ7zPOc8s3znnOfMOaKqGGOMMd1BSqIzYIwxxvhlQcsYY0y3YUHLGGNMt2FByxhjTLdhQcsYY0y3YUHLGGNMt2FByyQtEdkiIhd2wnF+LyI/iNO+bhSRpW3ZVkQOiciIeOTDmK7KgpYxjoiUiMhTIrJXRKpE5F0RuTHR+fJLVfuo6qZE58OYjpSW6AwYkwgikqaqwYjFfwDeAYYCdcB4YGBn580Y0zyraZkeQUS+IyJ/FpE/ikgAuDHKZmcBv1fVw6oaVNWVqroobB9ni8hrInJQRLZF1MIKROR5EakWkTdE5OSwdKeKyAsisl9ENojI1WHrCkVkgYgERGQ5EJ5umIioiKSFLXtRRG5upowqIiPd49+LyNwW8nSRy0uViPxaRF5qbr/GdCUWtExPMhP4M5APPBZl/TJgrojMEpEh4Svc80XAL4EiYAKwKmyTa4DvAgVAOXC3S5cNvAA8DvR32/1aRMa6dHOBWqAY+Jy7xUtzeeqHdx7uBAqBDcBHwsvqAvOQE/ZoTIJZ0DI9yeuq+oyqNqpqTZT1VwGvAHcBm0VklYic5dZdB/xDVZ9Q1QZV3aeqq8LS/kVVl7smx8fwghrApcAWVX3E1d7eBp4CPiUiqcCVwH+72t0a4NE4lre5PF0MrFXVv7h19wG7QolUdauq5qvq1jjmxZi4sKBlepJtLa1U1QOqeoeqjgUG4NWknhERAQYDG1tIvivs8RGgj3s8FJjsai4HReQgXgAciFdjS4vI14f+ixNTc3k6KfyY6o2aXRHH4xrTYSxomZ7E95QGqroX+CneF3xfvC/5k1tMFN024CVXcwnd+qjqF4FKIIgXEEPCm+QOu/veYcvi0TFkJ1ASeuKCcknzmxvTdVjQMsYRkR+JyDgRSRORHOCLQLmq7sNrXrtQRK526wtFZIKP3T4HjBaR60Uk3d3OEpExqnoU+AvwHRHpLSKlwA2hhKpaCWwHPiMiqSLyOdoWOCM9D4wXkctdJ48vY70kTTdhQcuYY3oDTwMHgU14TXuXgXedB+9a0NeA/XhNh6fH2qGqVgMXAbOAHXhNdj8CMt0mt+I12+0Cfg88ErGLzwP/CewDxgKvtalkx+dpL971ux+7/ZYCK/C6+Yc6YhyyjhimKxKbBNKYnk1EUvCuaV2nqv9KdH6MaYnVtIzpgURkmojki0gm8E1A8Lr8G9OlWdAypmeaitcbci/wCeDyZv4GYEyXYs2Dxhhjug2raRljjOk2LGgZY4zpNixodSARuVtEXnEDtfaOWDdMRCrdAKgvikhRovJpjDHdhQWtDiIi44CTVfUc4B9EHwj1JVX9qLtVdm4OTSLZDxqTbEQkT0SWu//4jYuyPi7vawtaHeccvFHBcfdnR9nm39wX1z1uKB3TA9gPGpOkjgCX4M0g0Jx2v68TFrTcfEQTEnX89hCR/xaRX8XYrACoco+r8MavC7cTGAmcizdlxRVxzWQHEJG1IvLRVmz/QxH5jw7LUPdlP2hM0nGzH8QKRCe8r13tbGyMdE0SErREpABvrLP17djH9gQGvVLgXREZKCJLo9wGAgeAPLd9Ht7QP01Utc5NR6F448/FHBIo0VR1rKq+6GdbV/WfDTzoc/tEvp6drVN/0PSwc9tjdYPXubn39U+B7/ndSVrsTTrEeOB9Va2LtlJEUt1golG5Sez6046g105jgV+o6i6i/0pGRJbiTbL3KDANeDVifa6qBtzTc0hcWTrKjcBCP39Y7QKvZ9y5Hy7Rmkk+hY8fNBwbB/AvwBS8Objako+kO7eJEOs7KdE663Vu6X3tvg+b1cL7egHwgIgUq+rOmJlQ1U6/4Q0SOj/s+c3A34Hf4n2g/xPvBVgA7AYCwF+BXLxIXQMcBQ7hDfiZ5vbzeWAd3q/XRUD/OOQ1BS/4bMUb8HSWO36Oj7Q/xJtU8Ckg2y0biDeb7CeAt9z6eaEyJOC12OLO92q8qTB+izeX1CKgGu+aS0HYthdGpP1/Lm0V8Ccgy637J/CZiGONwBv1fK/b/oW2vJ5481G95o63C2/6jxk+yjoHb4TzuS4PO4CPh62/G/h52PMSd05S3Ht0MXC/e4++j1fj/op7b+wFrvB5zscDj7vHtwD/HrE+N+I9NDvG/k44r2551HPb0uekLee2Pec17PMfl3Pr9ncIGByWNwUGuOf/CTzsHkf9jmnhO6lV+Yzxfmrre7hTPkPNvad8nv/fA+OiLG/2fe3KcYOv/bf2Sy4eN+Ah4Jthz+/D+4K8zL2gme6F+Lh73Bd4HfhPt/2XgT9F7PObeEFgJJABPAw8FLHNc3gjeEe7PddMXr+DV0saiver+FVgUyLOWwe9FlvwxpwbAAwC9gBvA2e4c/9P4Nth20YGreUcm3NqPTDHrasEzoo41lL32qUCWcC/teX1BO7F+5BeDaTjBc4PfZT113i1mmnuffZtvNmIQ+ufBz4b9vwSYHnYe3QfcIHL/1N4EzZ+xeXhK8DbrTjvx/2gwf2Yceta9YOmufMa7dzG+py05dy257x20LndhhdMBHgX+AA41T0vB05327X0HRPtO6lV+Yzxfmrre7hTPkMtvadi5G8h3o+W1/FaW3y9r925/ZmvY7TmCy5eN1egS8Oev4T7YmwhzfeAe9zjh4A7w9b1x+u5Mjps2VRgZTvzWYT3y+XkiDfCM4k4b+0syz+A70RZvgVvdO/Q86eA+8Oe/3uovEQPWp8Je/5j4AH3uAE4NeJYO4HbgIyI5a16Pd2XwT0R2yuultfCOVgKfD3s+VUc/+W6HTgz4rUO/Sp/Cfh/Yeu+T9gPHbwvstUJem2jntfIc+vnc9KWc9ue89qec9vCe/pdYBJeEH3a3aYAM/B6rzVXjvDvmBO+k1qbzxjvp7a+hzvlM9TSe6qD3sN3A7/zs22nd8RwPUbG4b2xQk4D/i9iu6tE5FUR2eOmKL8DrzoOMAF4J2zzC/B+SSwPm9L8bxy72N1WFwDrVTV8mvUBEXn3RUQ+JiLD2pmf0L6+KSIXtrD+f0WkLGLxI3i1pmh2hz2uifK8D81rbkr3A0BOxLbXATOBHSLyWxEJdUCYQOtez/Ec367eHzikqrUt5DOU7q9hz8fhNZ2EOo4MANaGrT89LF+n4dXUQ0qjPH8vxvE7SnPnFY4/t34+J205t+05r9D2c9vce/oA3vvwP4Bf4DX9FQBfwvtFj8tbS98xJ3wntSafPsrd1vdwZ32GWnpPdYQcvBavmBLRe3A40KiqHwKIyFC86mnTm1JEzsebKO8/8Jqe+uE1W61yc/+Mw5uEL6Qv8LQeP6V5nqp+NPzAIrLI/fEt2m0RJwodN5Q+Hbgc7xpOa30Or3kiHsbScuAs5fgPC6r6mKq+HKfj+7EaGB2Rh3+q6gUuf6cDN7b29RSRfLzp6cO71n6KY13IoxKR4XjXczaELT4j7LhjgQ9CH1rxZvT9GLDavUczOPaFBt6XRHieT4t43mminVdomicr/Ny2+Dlpy7ltz3l1z9t8blt4Tx8EyoBi9Xq7BvDOyzjgGXfclr5jon0ntTafLb2f8mnDe9iVuVM+Q829pzrQGI4Pus1KRNA6jeO/cE8H3lXVxohl2/DeNAXA7/B+CawDerlbeN7fBj4mIhPB65knIjMj/9+iqjNUtU8ztxlR8roBOFtERotIHt4F2CGh/Is3hfo/RWSFiHzMLcsQkTdEpJeIXCYiD4jIZXjtuY+IyGwReVtEfi4i690+HhSR90XkErePL4vIMrefqW7ZF0XkNRH5Ld6HcXe04zu9NPHTTCwEzgs9EZErRGSUe01y8F7XVbT+9RyPd8H5WvGmvb8E7xf0d2Lk5zROfJ+dwbEPigC93T5T8Jo6i/C+XI97j4pILt41zvAfL5G1h7gTkd+LyO8jljV3XuHEcxvrc9KWc9ue8wodc24PAF/lWK0qgHe96UE91gOwpe+Y5r6TWpPPlsrdpvdwZ32GYryn4k68Od3OxOuMEVMiuryP58QXelXENo/htYvvAtbgVcHXqWo9UC8iDwDrRCSgqiWq+rqIfA94Srxq+UHgr6r6bHsyqqoviMh8vKnId+Fd76kFPhBvVIMZeNXwHOAJ4F+qWi8iv8Frlihy5WjEa0v+mHhdU78N3IV3MfKPwFl4H/5ZIlKJ141+Kl6A/IWIfAsvAPyb2/aHzR1fRPpzfPNeoszD+9UaCqBnA7/C6wG6HbhXVf8J0JrXU0TG470/puJ9OW3AmwtqXYz8HPdrXUQK8S4Sr3GLXsF7X76Hd3H9r0CFqh4Qkcj36GlAuaoecfuK9ku3IwwG5kcsa/a8qurhyHMb43PSlnPb5vPq1nfEuT2A9932uHseAPKB34Rt0+x3TJTXuy35bOn91Nb3cGd9hpo9Tge5DHhRVXf42djm02ojEbkLuJJj7bA7VPVat64UrzY2RFW3i8ho4HZVnSNes8T5qvotEbkI+IiqfkdEPoHXnFYILFLVV0RkMN4vtHXAq6r6T/GuVX0GrxfTCccXkQvwevr4/rNeRxGRe4A9qvrzOO7zfrz/+P1vvPbZHYhIBt6v+NNUtaGDjtEjz21nS/R5TvTxI4nIG8BNqrom5sYk7s/FySAfuFVVl0JTm3Xol+Z9wH/hXcz8McdfSD4NWOken47XBTT0+A3gIry2c/D+z/YsXs+ndLfsP4B/uX2ecHxOrMkmjKp+swN2Ox7vnPQorpVhTAcfpkee2wRI9HlO9PGPo6qTW7O9Ba22exD4nYg04HXvni0iB/Caxb6K95+lV0Xk13gXb38gXu/BHI71tDodr1kPvDfSb4DNwKMi8n3gFVWdLyIfAH8Uka147e734TU/HHd8vKaO8Xh/mExW42iml56IDMH1WouiVFW3dliukkPUc2vnNe4S/R5u9vjdgTUPGmOM6TZsahJjjDHdhgUtY4wx3UbSX9Pq16+fDhs2LNHZMMaYbuWtt97aq6pdbtbspA9aw4YNY8WKFYnOhjHGdCsi8mGi8xCNNQ8aY4zpNixoGWOM6TYsaBljjOk2fF3TEpHpeGPppeLNB3NvxHpx6y/Gm57iRlV9u6W0IvITvEFk64GNeJOlHXTr7gRuwhvU8TZVXeyWn4k3K2YvvAFZv6L2RzNjepyGhgYqKiqorY01k4eJJSsri5KSEtLT02Nv3AXEDFoikoo3jfbHgQrgTRFZEDG44wxglLtNxhsNfXKMtC/gTVoWFJEf4U1p/w03bt8svKH9TwL+ISKj3ejM9+NNT74ML2hNx8dw/saY5FJRUUFOTg7Dhg1Djp/MwbSCqrJv3z4qKioYPnx4orPji5/mwUl4oxlvcuOfzcebHCzcTGCeepYB+SJS3FJaVf27qgZd+mVASdi+5qtqnapuxpsee5LbX66qvu5qV/Pw5rYyxvQwtbW1FBYWWsBqJxGhsLCwW9VY/QStQXjzzoRUuGV+tvGTFrwJEkM1ppb2VeFjX4jILeLNMbWisrIy2ibGmG7OAlZ8dLfz6CdoRStR5HWk5raJmVZE/gsI4s3v0q59NS1UfUhVy1S1rKioc/4bt2FXNcs27euUYxljTE/lJ2hV4E0+F1ICRE7W1dw2LaYVkRuAS4HrwjpUtLSvkijLu4RpP3+ZWQ8tS3Q2jDGdYNu2bXzsYx9jzJgxjB07ll/84hcAfOc732HQoEFMmDCBCRMmsHDhwqY0q1evZurUqYwdO5bx48e32CT305/+FBFh7969Tct++MMfMnLkSE455RQWL17ctPytt95i/PjxjBw5kttuu43QV2ldXR2f/vSnGTlyJJMnT2bLli1xPgsJoqot3vA6a2wChuPN8/QOMDZim0vwmvcEb+6n5bHS4nWiWAcURexrrNsu06XbBKS6dW+6/Ys73sWx8n/mmWdqZxj6jed06Dee07qGo51yPGN6snXr1iX0+Dt27NC33npLVVUDgYCOGjVK165dq9/+9rf1Jz/5yQnbNzQ06Pjx43XVqlWqqrp3714NBoNR971161a96KKLdMiQIVpZWamqqmvXrtXTTjtNa2trddOmTTpixIim9GeddZa+9tpr2tjYqNOnT9eFCxeqqurcuXP1C1/4gqqqPvHEE3r11Vc3W55o5xNYoTG+XxNxi1nTUq+zxK3AYrw5op5U1bUiMkdE5rjNFrrgUo43J9SXWkrr0vwKb26pF0RklZsuGrf+SRfQ/gZ8Wb2egwBfBB52x9lIF+w5uP1gTaKzYIzpYMXFxUycOBGAnJwcxowZw/bt25vd/u9//zunnXYap59+OgCFhYWkpqZG3farX/0qP/7xj4+71vTss88ya9YsMjMzGT58OCNHjmT58uXs3LmTQCDA1KlTERFmz57NM88805TmhhtuAOBTn/oUS5YsaaqFdWe+/qelqgvxAlP4sgfCHivwZb9p3fKRLRzvbuDuKMtX4E1g1mVt3X+E4f2yE50NY3qM7/51Let2BOK6z9KTcvn2J8b62nbLli2sXLmSyZMn8+qrr/KrX/2KefPmUVZWxv/8z/9QUFDA+++/j4gwbdo0KisrmTVrFl//+tcBuPnmm5kzZw5lZWUsWLCAQYMGNQW3kO3btzNlypSm5yUlJWzfvp309HRKSkpOWB5KM3iwd6UlLS2NvLw89u3bR79+/dp1bhLNRsSIs637Dic6C8aYTnLo0CGuvPJKfv7zn5Obm8sXv/hFNm7cyKpVqyguLuZrX/saAMFgkKVLl/LYY4+xdOlSnn76aZYsWQLAww8/TFlZGUeOHOHuu+/me9/73gnHiVZDEpFml7eUprtL+lHeO4OqIgKqXk3LGNN5/NaI4q2hoYErr7yS6667jiuuuAKAAQMGNK3//Oc/z6WXXgp4NaDzzjuvqZZz8cUX8/bbb3PBBRc0bb9x40Y2b97cVMuqqKhg4sSJLF++nJKSErZtO/ZPoIqKCk466SRKSkqoqKg4YXnomNu2baOkpIRgMEhVVRV9+/btoLPReaymFQeH648S+lHz4T4LWsYkO1XlpptuYsyYMdx+++1Ny3fu3Nn0+Omnn2bcOO9qxrRp01i9ejVHjhwhGAzy0ksvUVpaetw+x48fz549e9iyZQtbtmyhpKSEt99+m4EDB3LZZZcxf/586urq2Lx5Mx988AGTJk2iuLiYnJwcli1bhqoyb948Zs70xn647LLLePTRRwH485//zPnnn281LeMJ1DQ0PbaaljHJ79VXX+UPf/gD48ePZ8KECQDcc889PPHEE6xatQoRYdiwYTz44IMAFBQUcPvtt3PWWWchIlx88cVccsklwPHXtJozduxYrr76akpLS0lLS2Pu3LlNHTnuv/9+brzxRmpqapgxYwYzZswA4KabbuL6669n5MiR9O3bl/nz53fgGek8kgy9SVpSVlamHT0J5IZd1Uz7+csMyM2kujbI2u9OS4pfNMZ0VevXr2fMmDGJzkbSiHY+ReQtVW0+kiaINQ/GQaDWq2mNOymPI/VH2XuoPsE5MsaY5GRBKw5CzYPjBuUB1kRojDEdxYJWHDTVtFzQ2rLXur0b09GS/dJGZ+lu59GCVhxU13ozrIwflEdairCx8lCCc2RMcsvKymLfvn3d7gu3q1E3n1ZWVlais+Kb9R6Mg1DzYEF2OkMLe1vQMqaDhf6fZFMPtV9o5uLuwoJWHARqg2SmpZCZlsrJRX0o32NBy5iOlJ6e3m1m2jXxZc2DcRCoaSC3VzoAI/v34cN9R2g42pjgXBljTPKxoBUHgdoGcrO8SuvJRX0INqqNjGGMMR3AglYcVNcGj6tpAdZEaIwxHcCCVhwEahrIyfKC1ogib1oS64xhjDHxZ0ErDgK1wabmwZysdAbmZrHRalrGGBN3voKWiEwXkQ0iUi4id0RZLyJyn1u/WkQmxkorIleJyFoRaRSRsrDl17mZjEO3RhGZ4Na96PYVWte/XaWPk/COGAAn98+m3GpaxhgTdzGDloikAnOBGUApcI2IlEZsNgMY5W63APf7SLsGuAJ4OXxHqvqYqk5Q1QnA9cAWVV0Vtsl1ofWquqcVZe0Qqupd08o6FrROGZDL+7urOdpof3w0xph48lPTmgSUq+omVa0H5gMzI7aZCcxTzzIgX0SKW0qrqutVdUOMY18DPNGK8nS6umAj9Ucbye117C9vY4pzqG1oZLMN52SMMXHlJ2gNAraFPa9wy/xs4ydtSz7NiUHrEdc0eJc0M/+HiNwiIitEZEVH/2M+NBpGTlhNa0xxLgDrdwY69NjGGNPT+Ala0QJDZLtXc9v4SRv9oCKTgSOquiZs8XWqOh44x92uj5ZWVR9S1TJVLSsqKvJzuDYLDZYb6ogBMGpAH9JSxIKWMcbEmZ+gVQEMDnteAuzwuY2ftM2ZRUQtS1W3u/tq4HG85seECrjBcsM7YmSmpTKyfx8LWsYYE2d+gtabwCgRGS4iGXjBZEHENguA2a4X4RSgSlV3+kx7AhFJAa7CuwYWWpYmIv3c43TgUrzOHAkVah4M74gBXhPhOgtaxhgTVzGDlqoGgVuBxcB64ElVXSsic0RkjttsIbAJKAd+A3yppbQAIvJJEakApgLPi8jisMOeC1So6qawZZnAYhFZDawCtrtjJVRTTSvr+LGHxxTnsDtQx/7DNouxMcbEi69R3lV1IV5gCl/2QNhjBb7sN61b/jTwdDNpXgSmRCw7DJzpJ7+dqamm1evEmhZ4nTH+bWS/Ts+XMcYkIxsRo52qm2paxwetUhe01myv6vQ8GWNMsrKg1U6B2gbSU4Ws9ONPZWGfTAb37cU7FQcTkzFjjElCFrTaKTRYbrS/jJ1eks+qrQc7P1PGGJOkLGi1U/hguZEmDM5nR1UtewK1nZwrY4xJTha02qm6tuGEThghEwbnA7Bq28HOy5AxxiQxC1rtFKhpOKETRsjYk/JITRG7rmWMMXFiQaudArVBcpppHuyVkcqpA3OspmWMMXFiQaudWqppAZw+OJ/V26potGlKjDGm3SxotVN1bfC4aUkiTRxSQHVdkA27qzsxV8YYk5wsaLVDfbCRmoajLda0Jg/vC8Dyzfs7K1vGGJO0LGi1Q3Vt9CGcwpUU9OKkvCze2Lyvs7JljDFJy4JWO4QGy22uIwaAiDB5RCHLN+/HG6LRGGNMW1nQaofmpiWJNHl4X/Yeqmdj5eHOyJYxxiQtC1rtUB1lAshoJtl1LWOMiQsLWu0QaLqm1fIML8P7ZVOUk2nXtYwxpp0saLVDqHkwJ0bzoIgwdUQhr5bvs/9rGWNMO/gKWiIyXUQ2iEi5iNwRZb2IyH1u/WoRmRgrrYhcJSJrRaRRRMrClg8TkRoRWeVuD4StO1NE3nX7uk+iDa3eiZpqWi10xAg5d3QRew/VsX5XoKOzZYwxSStm0BKRVGAuMAMoBa4RkdKIzWYAo9ztFuB+H2nXAFcAL0c57EZVneBuc8KW3+/2HzrWdD+F7CjVtUFSBLIzfAStUd7sxS+/v7ejs2WMMUnLT01rElCuqptUtR6YD8yM2GYmME89y4B8ESluKa2qrlfVDX4z6vaXq6qvq9d3fB5wud/0HSE0l1ZKSuwKX//cLMYU5/LS+3s6IWfGGJOc/AStQcC2sOcVbpmfbfykjWa4iKwUkZdE5JywY1T42ZeI3CIiK0RkRWVlpY/DtU1Lg+VGc+7ofrz14QEO1QU7LE/GGJPM/AStaNWIyN4EzW3jJ22kncAQVT0DuB14XERyW7MvVX1IVctUtayoqCjG4dou1mC5kc4bXUTDUeX1jdaL0Bhj2sJP0KoABoc9LwF2+NzGT9rjqGqdqu5zj98CNgKj3b5KWrOvjhZrsNxIZUP7kp2RyosbrInQGGPawk/QehMYJSLDRSQDmAUsiNhmATDb9SKcAlSp6k6faY8jIkWuAwciMgKvw8Umt79qEZnieg3OBp71X9T4C9S2rqaVkZbCuaOLeGHdbuv6bowxbRAzaKlqELgVWAysB55U1bUiMkdEQj37FgKbgHLgN8CXWkoLICKfFJEKYCrwvIgsdvs6F1gtIu8AfwbmqGpoKIkvAg+742wEFrWn8O0V6ojRGtPGDmRPdR2rbDZjY4xpNV9tW6q6EC8whS97IOyxAl/2m9Ytfxp4Osryp4CnmtnXCmCcnzx3hkArmwcBPnZqf9JShMVrdjFxSEEH5cwYY5KTjYjRRkcblUN1wVY1DwLk9Upn6smFLF67y0Z9N8aYVrKg1UaHfA6WG820sQPZsu8I7+8+FO9sGWNMUrOg1UatGcIp0kWlAxCBRWt2xjtbxhiT1CxotVGVz8Fyo+mfm8Xk4X1ZsGqHNREaY0wrWNBqo2NzabW+pgXwyTMGsWnvYVZXVMUzW8YYk9QsaLXRsebB1te0AKaPKyYjLYWnV26PZ7aMMSapWdBqo9BcWnlt6IgRSnfhmP48t3oHwaON8cyaMcYkLQtabRRwzYOtGTA30swJg9h7qJ5Xym26EmOM8cOCVhtVu+bBPpltD1ofPaWIgt7p/HlFReyNjTHGWNBqq0BNkD6ZaaSltv0UZqalcuXEEhav3cWe6to45s4YY5KTBa028gbLbXstK+SayUMINir/Z7UtY4yJyYJWG7VlsNxoTi7qw9QRhTyxfKuN/G6MMTFY0Gqj1s6l1ZJrJw+h4kANL3/QcbMsG2NMMrCg1UatnUurJdPGDqRfnwzmvf5hXPZnjDHJyoJWGwVqG9o0WG40GWkpXD9lGP98bw/le6rjsk9jjElGFrTaKFATbNd/tCJdP3UomWkpPPzK5rjt0xhjko2voCUi00Vkg4iUi8gdUdaLiNzn1q8WkYmx0orIVSKyVkQaRaQsbPnHReQtEXnX3Z8ftu5Ft69V7ta/7UVvO1WlOo7NgwB9szO4qqyEv7y93bq/G2NMM2IGLRFJBeYCM4BS4BoRKY3YbAYwyt1uAe73kXYNcAXwcsS+9gKfUNXxwA3AHyLWX6eqE9xtj69Sxtnh+qM0atsHy23OTWePoKGxkXmv2bUtY4yJxk9NaxJQrqqbVLUemA/MjNhmJjBPPcuAfBEpbimtqq5X1Q2RB1PVlaq6wz1dC2SJSGabStdBQuMOxrOmBTC8XzbTSgfy6OtbmqY+McYYc4yfoDUI2Bb2vMIt87ONn7QtuRJYqap1YcsecU2Dd4mIREskIreIyAoRWVFZGf9u5E0jvMepI0a42y4YRXVtkN8utWtbxhgTyU/QihYYIv8F29w2ftJGP6jIWOBHwBfCFl/nmg3Pcbfro6VV1YdUtUxVy4qKivwcrlUCNe0fLLc5pSflMmPcQH63dDMHj9THff/GGNOd+QlaFcDgsOclwA6f2/hJewIRKQGeBmar6sbQclXd7u6rgcfxmh87XXU759KK5SsXjuJQXdB6EhpjTAQ/QetNYJSIDBeRDGAWsCBimwXAbNeLcApQpao7faY9jojkA88Dd6rqq2HL00Skn3ucDlyK15mj03Vk8yDAqQNzueS0Yh55dTN7D9XFTmCMMT1EzKClqkHgVmAxsB54UlXXisgcEZnjNlsIbALKgd8AX2opLYCIfFJEKoCpwPMistjt61ZgJHBXRNf2TGCxiKwGVgHb3bE6Xah5MB4D5jbn9o+Ppi7YyM9eeL/DjmGMMd2Nr29dVV2IF5jClz0Q9liBL/tN65Y/jdcEGLn8B8APmsnKmX7y29FCvQfjMWBuc04u6sNnpgxl3utbuGHqME4ZmNNhxzLGmO7CRsRog+q6IFnpKWSkdezp+8oFo8jJSufuhes79DjGGNNdWNBqg0BNfEfDaE5Bdga3XTCKl9+v5F/vJeR/1MYY06VY0GqDeA6WG8v1U4Yyoiib/16whpr6o51yTGOM6aosaLVBvAfLbUlGWgr3fHI82/bX8IslH3TKMY0xpquyoNUG8R4sN5YpIwq5uqyEh1/ZxHu7Ap12XGOM6WosaLVBoDbYac2DIXfOGENur3TueOpdjjb6GlTEGGOSjgWtNvA6YnRO82BIQXYG3/5EKau2HeTBlzfGTmCMMUnIglYrqSqB2oYO/Y9Wcy47/SQuOa2Y/33hfdZsr+r04xtjTKJZ0GqlumAjDUc17nNp+SEi3H35OAp6Z/DVP62itsF6ExpjehYLWq3UUXNp+ZXfO4OfXnU6H+w5xA/tT8fGmB7GglYrdfRguX6cO7qIm84ezqOvf8hf34k5aL4xxiQNC1qtVNUJg+X6cceMUzlzaAF3PLWa8j3VCc2LMcZ0FgtarRSaSysRHTHCpaemMPfaiWSlpzLnj29zuC6Y0PwYY0xnsKDVSoFaLzjkJaAjRqSBeVn88poz2FR5iK/+aRWN9v8tY0ySs6DVSonuiBHpIyP7cdelpfx93W7u/dt7ic6OMcZ0qMRXF7qZrtARI9KNHxnG5r2HeejlTQwrzObayUMSnSVjjOkQvmpaIjJdRDaISLmI3BFlvYjIfW79ahGZGCutiFwlImtFpFFEyiL2d6fbfoOITAtbfqaIvOvW3Sci0rZit12gJkh6qpDZwXNptYaI8N+XlvLRU4q469k1vLjBpjExxiSnmN+8IpIKzAVmAKXANSJSGrHZDGCUu90C3O8j7RrgCuDliOOVArOAscB04NduP7j93hJ2rOmtKGtchAbLTUC8bFFaagq/vOYMTh2Yw5w/vsXyzfsTnSVjjIk7P9WFSUC5qm5S1XpgPjAzYpuZwDz1LAPyRaS4pbSqul5VN0Q53kxgvqrWqepmoByY5PaXq6qvq6oC84DLW13idkrEYLl+5WSl8+jnJnFSfi8+9/s3ebfChnoyxiQXP0FrELAt7HmFW+ZnGz9p/R5vkHscc18icouIrBCRFZWVlTEO1zqJGCy3Nfr1yeSxmyeT1yud2b97g/d323+4jDHJw0/QitYOFtm3urlt/KT1ezzf+1LVh1S1TFXLioqKYhyudRI1WG5rFOf14vHPTyY9NYVZDy1j7Q6rcRljkoOfoFUBDA57XgJEjh3U3DZ+0vo9XoV73Jp9xV11bTAhg+W21tDCbP70halkpaVwzUPLWLXtYKKzZIwx7eYnaL0JjBKR4SKSgddJYkHENguA2a4X4RSgSlV3+kwbaQEwS0QyRWQ4XoeL5W5/1SIyxfUanA0867eg8eI1D3btmlbI8H5e4MrvncFnHn6DN7dY5wxjTPcWM2ipahC4FVgMrAeeVNW1IjJHROa4zRYCm/A6TfwG+FJLaQFE5JMiUgFMBZ4XkcUuzVrgSWAd8Dfgy6oamoPji8DD7jgbgUXtK37rBWobumxHjGgG9+3Nk1+YSv/cTD7z8BssendnorNkjDFtJl5HvORVVlamK1asiMu+6oONjP7WIr728dH8+wWj4rLPzrL/cD03P/omK7cd5FuXlHLT2cMTnSVjTBcmIm+palnsLTtX1/mHbDdQ3QVHw/Crb3YGj39+CtPHDuT7z63jOwvWctTGKjTGdDMWtFohNFhud+iIEU1Weipzr53IzWcP5/evbeHGR5Zz4HB9orNljDG+WdBqha42WG5bpKQI37q0lHuvGM8bm/bziV8ttS7xxphuw4JWKwS6yFxa8TBr0hD+9IUpBI8qV97/Gk+vrIidyBhjEsyCVitUd/PmwUhnDCngr/9+NqeV5PPVP73D1558h0M2maQxpguzoNUKydA8GKkoJ5PHb57MbeeP5OmVFVx63yu8Y39ENsZ0URa0WqErzqUVD2mpKdx+0Sk88fkp1AcbufL+15j7r3KCRxsTnTVjjDmOBa1WCNQESRHIzkiNvXE3NHlEIYu+ci7Txg7kJ4s38Mlfv8Z7uwKJzpYxxjSxoNUK1W6w3K42l1Y85fVO51fXnsHcayey42ANn/jlUv73hfepD1qtyxiTeBa0WiHQTQbLbS8R4ZLTinnh9vO4eHwxv1jyAZ/45VLe2LQv0VkzxvRwFrRaoTsNlhsPfbMz+MWsM3h4dhmH6oJ8+qFlfGX+SnYHahOdNWNMD2VBqxUCtT0raIVcWDqAf9x+HredP5JFa3Zx/k9f5MGXNlqToTGm01nQaoXq2iA5XXjW4o7UKyOV2y86hRe+ei5TRhTyw0XvceHPXmLBOztotDEMjTGdxIJWKwRqute0JB1haGE2v73xLB757Fn0zkjltidWMnPuq7xavjfRWTPG9AAWtFohUBvskc2D0XzslP48f9s5/Ozq09l/uJ7rHn6D63/7Biu3Hkh01owxScyClk/Bo40cqusZvQf9Sk0RrphYwpKvnce3LhnDmu1VfPLXr3H9b22WZGNMx/AVtERkuohsEJFyEbkjynoRkfvc+tUiMjFWWhHpKyIviMgH7r7ALb9ORFaF3RpFZIJb96LbV2hd/3afAZ9CY/Ilw2C58ZaVnsrN54xg6TfO584Zp7JuR4CrHnidax5axmsb95LsE40aYzpPzKAlIqnAXGAGUApcIyKlEZvNAEa52y3A/T7S3gEsUdVRwBL3HFV9TFUnqOoE4Hpgi6quCjvWdaH1qrqn9UVum6bBcntoRww/sjPT+MJ5J7P0G+fzrUvGUF55iGt/8waXz32VBe/soMGGhTLGtJOfmtYkoFxVN6lqPTAfmBmxzUxgnnqWAfkiUhwj7UzgUff4UeDyKMe+BniiNQXqKFU1yTnuYEfoleHVvF75+sf4/uXjCNQGue2JlZz343/x4Esbm86lMca0lp+gNQjYFva8wi3zs01LaQeo6k4Adx+tqe/TnBi0HnFNg3dJM+MpicgtIrJCRFZUVlY2X7JWaBos15oHfctKT+X6KUNZcvt5PDy7jKGF2fxw0XtM/eES/vvZNWzYVZ3oLBpjuhk/bV3RAkPkRYrmtvGTNvpBRSYDR1R1Tdji61R1u4jkAE/hNR/OO+EAqg8BDwGUlZXF5YJKoCZ0TcuaB1srJUW4sHQAF5YOYM32Kn63dDPzl29j3usfUja0gGsnD+Hi8cVkpSfnQMTGmPjxU9OqAAaHPS8BdvjcpqW0u10TIu4+8vrULCJqWaq63d1XA4/jNT92impX08qz5sF2GTcoj599egLLvnkB37z4VPYdruf2J99h8j1L+P5z6yjfY7UvY0zz/AStN4FRIjJcRDLwgsmCiG0WALNdL8IpQJVr8msp7QLgBvf4BuDZ0M5EJAW4Cu8aWGhZmoj0c4/TgUuB8FpYhwo0dcSwoBUPfbMzuOXck1ly+3k8fvNkzh7Zj0df28KFP3uZy361lEde3cy+Q3WJzqYxpouJ2dalqkERuRVYDKQCv1PVtSIyx61/AFgIXAyUA0eAz7aU1u36XuBJEbkJ2IoXpELOBSpUdVPYskxgsQtYqcA/gN+0rditF5q1uI81D8ZVSorwkZH9+MjIflRW1/Hsqu08vXI73/3rOu5+fj0fPaWIT55RwgVj+lvzoTEGSfb/0JSVlemKFSvavZ/v/nUtf15RwbvfnRaHXJlYNuyq5i8rK3hm5XZ2B+rIyUrj46UDuHhcMeeM7kdmmgUwYzqSiLylqmWJzkckqzb41JMHy02EUwbmcOeMMXx92qm8vnEfz67azt/X7eYvb28nJzONC8b05+LxxZw7ushqYMb0IPYt7JMNlpsYqSnC2aP6cfaoftwdbOS1jXtZ9O4uFq/bxTOrdpCdkcr5YwZw4Zj+fHR0f/J622tkTDKzoOVTT51LqyvJSEvho6f056On9OcHR8fx+sZ9LFqzk7+v3c1f39lBaopQNrSAC8b054IxAxjRL5tm/spnjOmmLGj5FKgJclJ+VqKzYZz01BTOHV3EuaOLuPtyZVXFQf65fg//WL+bexa+xz0L32NYYW8uGDOA80YXMWl4X2tGNCYJWNDyqbqugZysnERnw0SRkiJMHFLAxCEF/L9pp7D9YA3/XL+bJe/t4Q/LPuS3SzeTkZbCWcMKOHtkEeeM6kdpcS4pKVYLM6a7saDlU6AmaIPldhOD8ntx/dRhXD91GEfqgyzfvJ+lH+zllQ/28qO/vceP/ub9T+wjJxdyzqh+fOTkfpQU9LKmRGO6AfsW9qGxUamutY4Y3VHvjLSm62AAewK1LC3f6wWx8r08t3onACflZTFpeF8mDS9k0vC+nFxk18OM6YosaPlwuD5Io9poGMmgf24WV0ws4YqJJagq7+8+xBub9/HG5v0sLd/HM6u8Ucb69cnwgtgwL5CdMjCHVGtONCbhLGj5EJpLy/6nlVxEhFMG5nDKwBxmTx2GqrJ572GWb97P8s37eWPzfha+uwuA7IxUTh+czxlD8jljcAEThuTTr09mgktgTM9j38I+NE1LYs2DSU1EGFHUhxFFfZg1aQgAFQeOsHzzflZuPcjKbQd44KVNHG30RpEZ0rc3ZwzJZ+KQAs4Yks+pA3PJSPM1Gbgxpo0saPkQmpbEmgd7npKC3pQU9OaKiSUA1NQf5d3tVazceoCVWw+60Tq8JsWM1BROLc5h7El5jB+Ux7hBuZwyMMeGnDImjixo+RBomrXYTldP1ysj1XXY6AuAqrKzqpaVWw/yTsVB1myv4vnVO3hi+VYA0lKE0QNyGDcol/GD8hg7KI8xA3PplWGBzJi2sG9hH6rrvKCVYzUtE0FEOCm/Fyfl9+KS04oBL5BVHKjh3e1VrNlexZodAf6xfg9PrqgAIEVgWL9sxgzMbbqmdurAHAYX9Lb/jhkTgwUtH441D9rpMrGJCIP79mZw395cPP5YINtZVdsUxN7bGWDNjioWrtlJaKKF3hmpjBqQwxgXyLxglkvf7IwElsaYrsW+hX0INQ9aTcu0VXiN7KKxA5uWH6kP8v7uQ7y3M8B7u6rZsKuav6/bzfw3tzVt069PBicX9eHk/n0YWdSHkf29xyflZdl/yUyPY0HLh0BtA73SU61nmIm73hlpTBicz4TB+U3LVJXKQ3W8t9MLYh/sqWZj5WGeX72TKvcDykubyoiibEYW9eHksGA2rDDb3qsmafkKWiIyHfgF3ozBD6vqvRHrxa2/GG/m4htV9e2W0opIX+BPwDBgC3C1qh4QkWHAemCD2/0yVZ3j0pwJ/B7ohTdb8le0E2axDNQErROG6TQiQv+cLPrnZHHu6KKm5arK3kP1bKw8RPmeQ033b2450PSnaPCumQ0q6MWwwmyGFWYztLC397if1xPSBg423VnMb2IRSQXmAh8HKoA3RWSBqq4L22wGMMrdJgP3A5NjpL0DWKKq94rIHe75N9z+NqrqhCjZuR+4BViGF7SmA4taV+TW8wbLtaZBk1giQlFOJkU5mUwZUXjcusN1QTbvPdwUzLbsO8KH+w7zzKrtTX+O9/YBJ+X1Yli/3gwtzGZYYeg+myF9e1uvRtPl+ak+TALKVXUTgIjMB2YC4UFrJjDP1XqWiUi+iBTj1aKaSzsT+KhL/yjwIseC1gnc/nJV9XX3fB5wOZ0QtGywXNPVZWemMW5QHuMG5R23XFU5eKSBLfsOe7e9XjDbsu8Ii97dyYEjDcdt369PhvtvWi8G9/XuSwp6M7jAux5ntTSTaH6+iQcB28KeV+DVpmJtMyhG2gGquhNAVXeKSP+w7YaLyEogAHxLVV9x+6qIcowTiMgteDUyhgwZEqt8MQVqG6wHl+mWRISC7AwKsjM4Y0jBCeurwgLatv1HqDhQ09Rdf/HaXTQcPb71vX9OZlgw68Vg9+frQQW9KM7LsqBmOpyfoBWte1LkdaTmtvGTNtJOYIiq7nPXsJ4RkbGt2ZeqPgQ8BFBWVtbua16BmgaGFWa3dzfGdDl5vdM5vXc+p4d1BAk52qjsqa5l2/4aKg54AS0U2N7eeoDnVu9sGtIqpKB3OsV5vTgpP4uBeVlNj4vzvKA2MC/LRggx7eInaFUAg8OelwA7fG6T0ULa3SJS7GpZxcAeAFWtA+rc47dEZCMw2h2jJEY+OkR1bdAGyzU9TmqKuGDTq2kEkHDBo43sCnhBbcfBGnYFatlxsIadVbVUHKhhxYcHOBjR/AheE2RTQMvLojjfC2j9c7IYkJtJ/9ws+mTa581E5+ed8SYwSkSGA9uBWcC1EdssAG5116wmA1UuGFW2kHYBcANwr7t/FkBEioD9qnpUREbgde7YpKr7RaRaRKYAbwCzgV+2teB+qSoBm0vLmBOkpaY0jc3YnCP1QXZW1bLzYC07q7yAtrOqhh0Ha9m67wjLNu07rqNISHZGKgNysyjKyWRAbhb9Q/e5mV7PylzvuQW3nifmK66qQRG5FViM1239d6q6VkTmuPUP4PXkuxgox+vy/tmW0rpd3ws8KSI3AVuBq9zyc4HviUgQOArMUdX9bt0XOdblfRGd0AmjtqGRhqNqg+Ua0wa9M9K8P0YX9Wl2m0N1QXYerGFPdR17qmvZHahjd6DWex6o5Z2Kg+wO1FLb0HhC2uyMVPq7oNY/N4sBOZn0y8mkX59MCvtkUNTn2OP0VPvvWjKQTvibU0KVlZXpihUr2px+d6CWyfcs4e5PjuO6yUPjmDNjjF+qSnVdkD0BL6iFgtueQB27q2vZ44Jcc8ENIK9XOv36ZNDPBbLQ48LQ45xM+mVn0i8ng94ZVoMTkbdUtSzR+Yhkr0wM1bU2hJMxiSYi5Galk5uVzsj+Oc1up6ocrj/KvkN17D1UR2V1PfsO17G3up69h+qaHq/fFWBvdR2BKE2T4I02EqqhFWZnUNA7g76uF2bf7Az69j7+cU5Wmg123EksaMVQZYPlGtNtiAh9MtPok5nGUB89fuuCR9l/uL4pqHm3+qagt/dQPTsO1rJ2R4B9h+upD0avxaWmCAW90ykIBbOmoJZ+fMALe5ydkWpjR7aBfRPHYLMWG5O8MtNSm3pIxqKq1DR4Qe7A4Qb2H6ln/+E69h9u4MDhevYfqffuD9ezae8h9n/YwIEj9Sf8LSAkPVXI65VBfu908nqlk98rnbymx97y/N7p5Lp1+b0zyOuVTm5WGmk9+PqcBa0YmiaAtOZBY3o0EaF3Rhq9M9IoOfF/2lE1NnrX4kJBbf8hd3+4nqqaBg4eaaCqxnu8K1DLe7uqqapp4FBd9GbLkJysNC+49fYCXF5Y4AsFwbxeGZx/av+kGzzZglYMoe641jxojGmtlBRxASSdYfgfoKDhaCOBmgYO1jRQVdNA1ZEGDtbUuyDX0HTvPa5nR1WNt/2RBoJhNbv3vj+9I4qVUPZNHIM1DxpjOlt6agqFrmdja4Q6ohw84tXeknFYLQtaMQRqgmSkppCZZFVsY0zyCe+I4rcJs7uxb+IYvNEw0qyXjzHGdAEWtGKorg1aJwxjjOkiLGjFEKhpsMFyjTGmi7CgFYMNlmuMMV2HBa0YAjUN1jxojDFdhAWtGKprg+T2suZBY4zpCixoxRCobbDBco0xpouwoNWCuuBRahsabTQMY4zpIixotaBpCCfriGGMMV2Cr6AlItNFZIOIlIvIHVHWi4jc59avFpGJsdKKSF8ReUFEPnD3BW75x0XkLRF5192fH5bmRbevVe7Wv33Fb5kNlmuMMV1LzKAlIqnAXGAGUApcIyKlEZvNAEa52y3A/T7S3gEsUdVRwBL3HGAv8AlVHQ/cAPwh4ljXqeoEd9vTmsK2VqimZf/TMsaYrsFPTWsSUK6qm1S1HpgPzIzYZiYwTz3LgHwRKY6RdibwqHv8KHA5gKquVNUdbvlaIEtEWjdqZJzYYLnGGNO1+Alag4BtYc8r3DI/27SUdoCq7gRw99Ga+q4EVqpqXdiyR1zT4F3SzICAInKLiKwQkRWVlZUtl64FgaZZiy1oGWNMV+AnaEULDJFTcTa3jZ+00Q8qMhb4EfCFsMXXuWbDc9zt+mhpVfUhVS1T1bKioiI/h4vqWE3LmgeNMaYr8BO0KoDBYc9LgB0+t2kp7W7XhIi7b7o+JSIlwNPAbFXdGFquqtvdfTXwOF7zY4epdkHL/qdljDFdg5+g9SYwSkSGi0gGMAtYELHNAmC260U4BahyTX4tpV2A19ECd/8sgIjkA88Dd6rqq6EDiEiaiPRzj9OBS4E1rS1wawRqgqQIZGck30RqxhjTHcVs91LVoIjcCiwGUoHfqepaEZnj1j8ALAQuBsqBI8BnW0rrdn0v8KSI3ARsBa5yy28FRgJ3ichdbtlFwGFgsQtYqcA/gN+0p/CxhAbLtbm0jDGma/B1sUZVF+IFpvBlD4Q9VuDLftO65fuAC6Is/wHwg2aycqaf/MaLDZZrjDFdi42I0QIbLNcYY7oWC1otCNQ2kJNpNS1jjOkqLGi1IFBjNS1jjOlKLGi1IFBr17SMMaYrsaDVAu+algUtY4zpKixoNSN4tJFDdUEbLNcYY7oQC1rNOFRn4w4aY0xXY0GrGU2D5VrzoDHGdBkWtJrRNFiuNQ8aY0yXYUGrGQEbLNcYY7ocC1rNONY8aDUtY4zpKixoNeNY86DVtIwxpquwoNWM6lrriGGMMV2NBa1mBGq8mlafTGseNMaYrsKCVjO8wXLTSE2xubSMMaarsKDVDG+wXGsaNMaYrsRX0BKR6SKyQUTKReSOKOtFRO5z61eLyMRYaUWkr4i8ICIfuPuCsHV3uu03iMi0sOVnisi7bt190oFTClfXNtgQTsYY08XEDFoikgrMBWYApcA1IlIasdkMYJS73QLc7yPtHcASVR0FLHHPcetnAWOB6cCv3X5w+70l7FjTW19kfwK1DVbTMsaYLsZPTWsSUK6qm1S1HpgPzIzYZiYwTz3LgHwRKY6RdibwqHv8KHB52PL5qlqnqpuBcmCS21+uqr6uqgrMC0sTd4GaoI2GYYwxXYyfoDUI2Bb2vMIt87NNS2kHqOpOAHff38e+KmLkAwARuUVEVojIisrKyhYL15ypJxcyZURhm9IaY4zpGH6qEtGuG6nPbfyk9Xs83/tS1YeAhwDKyspiHS+quy6NbAE1xhiTaH5qWhXA4LDnJcAOn9u0lHa3a/LD3e/xsa+SGPkwxhiTxPwErTeBUSIyXEQy8DpJLIjYZgEw2/UinAJUuSa/ltIuAG5wj28Ang1bPktEMkVkOF6Hi+Vuf9UiMsX1GpwdlsYYY0wPELN5UFWDInIrsBhIBX6nqmtFZI5b/wCwELgYr9PEEeCzLaV1u74XeFJEbgK2Ale5NGtF5ElgHRAEvqyqR12aLwK/B3oBi9zNGGNMDyFeR7zkVVZWpitWrEh0NowxplsRkbdUtSzR+YhkI2IYY4zpNixoGWOM6TYsaBljjOk2LGgZY4zpNpK+I4aIVAIftjF5P2BvHLPTHViZe4aeVuaeVl5of5mHqmpRvDITL0kftNpDRFZ0xd4zHcnK3DP0tDL3tPJC8pbZmgeNMcZ0Gxa0jDHGdBsWtFr2UKIzkABW5p6hp5W5p5UXkrTMdk3LGGNMt2E1LWOMMd2GBS1jjDHdhgWtKERkuohsEJFyEbkj0flpLREZLCL/EpH1IrJWRL7ilvcVkRdE5AN3XxCW5k5X3g0iMi1s+Zki8q5bd5+bFgY3dcyf3PI3RGRYpxc0goikishKEXnOPU/28uaLyJ9F5D33Wk/tAWX+qntPrxGRJ0QkK9nKLCK/E5E9IrImbFmnlFFEbnDH+EBEQlNHdS2qarewG94UKhuBEUAG8A5Qmuh8tbIMxcBE9zgHeB8oBX4M3OGW3wH8yD0udeXMBIa78qe6dcuBqXgzRy8CZrjlXwIecI9nAX/qAuW+HXgceM49T/byPgrc7B5nAPnJXGZgELAZ6OWePwncmGxlBs4FJgJrwpZ1eBmBvsAmd1/gHhck+n1+wvlJdAa62s29yIvDnt8J3JnofLWzTM8CHwc2AMVuWTGwIVoZ8eY/m+q2eS9s+TXAg+HbuMdpeP+8lwSWsQRYApzPsaCVzOXNxfsCl4jlyVzmQcA296WaBjwHXJSMZQaGcXzQ6vAyhm/j1j0IXJOo17u5mzUPnij0wQipcMu6JVf1PwN4Axig3gzQuPv+brPmyjzIPY5cflwaVQ0CVUBhhxTCn58DXwcaw5Ylc3lHAJXAI65J9GERySaJy6yq24Gf4k0auxNvhvS/k8RlDtMZZewW330WtE4kUZZ1y/8FiEgf4CngP1Q10NKmUZZpC8tbStPpRORSYI+qvuU3SZRl3aa8ThpeE9L9qnoGcBiv2ag53b7M7jrOTLxmsJOAbBH5TEtJoizrVmX2IZ5l7BZlt6B1ogpgcNjzEmBHgvLSZiKSjhewHlPVv7jFu0Wk2K0vBva45c2VucI9jlx+XBoRSQPygP3xL4kv/wZcJiJbgPnA+SLyR5K3vKH8VKjqG+75n/GCWDKX+UJgs6pWqmoD8BfgIyR3mUM6o4zd4rvPgtaJ3gRGichwEcnAu1C5IMF5ahXXS+i3wHpV/VnYqgVAqEfQDXjXukLLZ7leRcOBUcBy1wxRLSJT3D5nR6QJ7etTwD/VNYR3NlW9U1VLVHUY3uv1T1X9DElaXgBV3QVsE5FT3KILgHUkcZnxmgWniEhvl9cLgPUkd5lDOqOMi4GLRKTA1Wovcsu6lkRfVOuKN+BivB53G4H/SnR+2pD/s/Gq9auBVe52MV679RLgA3ffNyzNf7nybsD1MnLLy4A1bt2vODaKShbwf0A5Xi+lEYkut8vXRznWESOpywtMAFa41/kZvB5fyV7m7wLvufz+Aa/XXFKVGXgC75pdA17t56bOKiPwObe8HPhsol/vaDcbxskYY0y3Yc2Dxhhjug0LWsYYY7oNC1rGGGO6DQtaxhhjug0LWsYYY7oNC1rGGGO6DQtaxhhjuo3/D6X3aw1ZbdcnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "optimizer = NoamOpt(\n",
    "    model_size=arch_args.encoder_embed_dim, \n",
    "    factor=config.lr_factor, \n",
    "    warmup=config.lr_warmup, \n",
    "    optimizer=torch.optim.AdamW(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9, weight_decay=0.0001))\n",
    "\n",
    "plt.title(\"lr Scheduling:\\n $lrate = d_{\\text{model}}^{-0.5}\\cdot\\min({step\\_num}^{-0.5},{step\\_num}\\cdot{warmup\\_steps}^{-1.5})$\")\n",
    "plt.plot(np.arange(1, 100000), [optimizer.rate(i) for i in range(1, 100000)])\n",
    "plt.legend([f\"{optimizer.model_size}:{optimizer.warmup}\"])\n",
    "None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#   &#x2728; 训练部分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T13:04:53.458868Z",
     "iopub.status.busy": "2023-03-12T13:04:53.458151Z",
     "iopub.status.idle": "2023-03-12T13:04:53.484067Z",
     "shell.execute_reply": "2023-03-12T13:04:53.481391Z",
     "shell.execute_reply.started": "2023-03-12T13:04:53.458821Z"
    }
   },
   "outputs": [],
   "source": [
    "from fairseq.data import iterators\n",
    "from torch.cuda.amp import GradScaler, autocast\n",
    "\n",
    "def train_one_epoch(epoch_itr, model, task, criterion, optimizer, accum_steps=1):\n",
    "    itr = epoch_itr.next_epoch_itr(shuffle=True)\n",
    "    itr = iterators.GroupedIterator(itr, accum_steps) # 梯度累积：更新每个accum_steps采样\n",
    "    \n",
    "    stats = {\"loss\": []}\n",
    "    scaler = GradScaler() # 自动混合精度`automatic mixed precision` (amp) \n",
    "    \n",
    "    model.train()\n",
    "    progress = tqdm(itr, desc=f\"train epoch {epoch_itr.epoch}\", leave=True)\n",
    "    for samples in progress:\n",
    "        model.zero_grad()\n",
    "        accum_loss = 0\n",
    "        sample_size = 0\n",
    "        # 梯度累积：更新每个accum_steps采样\n",
    "        for i, sample in enumerate(samples):\n",
    "            if i == 1:\n",
    "                # 清空CUDA缓存在第一部之后，可以有效减少 OOM 的可能\n",
    "                torch.cuda.empty_cache()\n",
    "\n",
    "            sample = utils.move_to_cuda(sample, device=device)\n",
    "            target = sample[\"target\"]\n",
    "            sample_size_i = sample[\"ntokens\"]\n",
    "            sample_size += sample_size_i\n",
    "            \n",
    "            # 混合精度训练`mixed precision training`\n",
    "            with autocast():\n",
    "                net_output = model.forward(**sample[\"net_input\"])\n",
    "                lprobs = F.log_softmax(net_output[0], -1)            \n",
    "                loss = criterion(lprobs.view(-1, lprobs.size(-1)), target.view(-1))\n",
    "                \n",
    "                # logging\n",
    "                accum_loss += loss.item()\n",
    "                # 反向传播\n",
    "                scaler.scale(loss).backward()                \n",
    "        \n",
    "        scaler.unscale_(optimizer)\n",
    "        optimizer.multiply_grads(1 / (sample_size or 1.0)) # (sample_size or 1.0) 处理零梯度的情况\n",
    "        gnorm = nn.utils.clip_grad_norm_(model.parameters(), config.clip_norm) # grad norm 裁剪，处理梯度爆炸\n",
    "        \n",
    "        scaler.step(optimizer)\n",
    "        scaler.update()\n",
    "        \n",
    "        # logging\n",
    "        loss_print = accum_loss/sample_size\n",
    "        stats[\"loss\"].append(loss_print)\n",
    "        progress.set_postfix(loss=loss_print)\n",
    "        if config.use_wandb:\n",
    "            wandb.log({\n",
    "                \"train/loss\": loss_print,\n",
    "                \"train/grad_norm\": gnorm.item(),\n",
    "                \"train/lr\": optimizer.rate(),\n",
    "                \"train/sample_size\": sample_size,\n",
    "            })\n",
    "        \n",
    "    loss_print = np.mean(stats[\"loss\"])\n",
    "    logger.info(f\"training loss: {loss_print:.4f}\")\n",
    "    return stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型验证 & 模型推理(预测)\n",
    "为了防止过度拟合，每个`epoch`都需要进行验证，以验证对未见数据的性能\n",
    "- 该程序本质上与训练相同，就是增加了推理（预测）步骤\n",
    "- 验证后，我们可以保存模型权重\n",
    "\n",
    "仅验证损失无法描述模型的实际性能\n",
    "- 基于当前模型直接生成翻译结果，然后使用参考译文（目标值）计算BLEU\n",
    "- 我们还可以手动检查翻译结果的质量\n",
    "- 我们使用fairseq序列生成器进行`beam search`以生成多个翻译结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:38:35.759605Z",
     "iopub.status.busy": "2023-03-12T12:38:35.759162Z",
     "iopub.status.idle": "2023-03-12T12:38:35.786486Z",
     "shell.execute_reply": "2023-03-12T12:38:35.785597Z",
     "shell.execute_reply.started": "2023-03-12T12:38:35.759569Z"
    }
   },
   "outputs": [],
   "source": [
    "# fairseq's beam search 生成器\n",
    "# 基于模型和给定输入序列, 通过beam search生成翻译结果\n",
    "sequence_generator = task.build_generator([model], config)\n",
    "\n",
    "def decode(toks, dictionary):\n",
    "    # 将Tensor装换成我们可阅读的句子(human readable sentence)\n",
    "    s = dictionary.string(\n",
    "        toks.int().cpu(),\n",
    "        config.post_process,\n",
    "    )\n",
    "    return s if s else \"<unk>\"\n",
    "\n",
    "def inference_step(sample, model):\n",
    "    gen_out = sequence_generator.generate([model], sample)\n",
    "    srcs = []\n",
    "    hyps = []\n",
    "    refs = []\n",
    "    for i in range(len(gen_out)):\n",
    "        # 对于每个栗子, 收集输入`input`, 翻译结果`hypothesis`和 参考`reference`（label）, 后续用于计算 BLEU\n",
    "        srcs.append(decode(\n",
    "            utils.strip_pad(sample[\"net_input\"][\"src_tokens\"][i], task.source_dictionary.pad()), \n",
    "            task.source_dictionary,\n",
    "        ))\n",
    "        hyps.append(decode(\n",
    "            gen_out[i][0][\"tokens\"], # 0： 表示使用 beam中最靠前的翻译结果\n",
    "            task.target_dictionary,\n",
    "        ))\n",
    "        refs.append(decode(\n",
    "            utils.strip_pad(sample[\"target\"][i], task.target_dictionary.pad()), \n",
    "            task.target_dictionary,\n",
    "        ))\n",
    "    return srcs, hyps, refs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T12:38:54.693599Z",
     "iopub.status.busy": "2023-03-12T12:38:54.692644Z",
     "iopub.status.idle": "2023-03-12T12:38:54.741645Z",
     "shell.execute_reply": "2023-03-12T12:38:54.740593Z",
     "shell.execute_reply.started": "2023-03-12T12:38:54.693534Z"
    }
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "import sacrebleu\n",
    "\n",
    "def validate(model, task, criterion, log_to_wandb=True):\n",
    "    logger.info('begin validation')\n",
    "    itr = load_data_iterator(task, \"valid\", 1, config.max_tokens, config.num_workers).next_epoch_itr(shuffle=False)\n",
    "    \n",
    "    stats = {\"loss\":[], \"bleu\": 0, \"srcs\":[], \"hyps\":[], \"refs\":[]}\n",
    "    srcs = []\n",
    "    hyps = []\n",
    "    refs = []\n",
    "    \n",
    "    model.eval()\n",
    "    progress = tqdm(itr, desc=f\"validation\", leave=True)\n",
    "    with torch.no_grad():\n",
    "        for i, sample in enumerate(progress):\n",
    "            # validation loss\n",
    "            sample = utils.move_to_cuda(sample, device=device)\n",
    "            net_output = model.forward(**sample[\"net_input\"])\n",
    "\n",
    "            lprobs = F.log_softmax(net_output[0], -1)\n",
    "            target = sample[\"target\"]\n",
    "            sample_size = sample[\"ntokens\"]\n",
    "            loss = criterion(lprobs.view(-1, lprobs.size(-1)), target.view(-1)) / sample_size\n",
    "            progress.set_postfix(valid_loss=loss.item())\n",
    "            stats[\"loss\"].append(loss)\n",
    "            \n",
    "            # 模型推理\n",
    "            s, h, r = inference_step(sample, model)\n",
    "            srcs.extend(s)\n",
    "            hyps.extend(h)\n",
    "            refs.extend(r)\n",
    "            \n",
    "    tok = 'zh' if task.cfg.target_lang == 'zh' else '13a'\n",
    "    stats[\"loss\"] = torch.stack(stats[\"loss\"]).mean().item()\n",
    "    stats[\"bleu\"] = sacrebleu.corpus_bleu(hyps, [refs], tokenize=tok) # 計算BLEU score\n",
    "    stats[\"srcs\"] = srcs\n",
    "    stats[\"hyps\"] = hyps\n",
    "    stats[\"refs\"] = refs\n",
    "    \n",
    "    if config.use_wandb and log_to_wandb:\n",
    "        wandb.log({\n",
    "            \"valid/loss\": stats[\"loss\"],\n",
    "            \"valid/bleu\": stats[\"bleu\"].score,\n",
    "        }, commit=False)\n",
    "    \n",
    "    showid = np.random.randint(len(hyps))\n",
    "    logger.info(\"example source: \" + srcs[showid])\n",
    "    logger.info(\"example hypothesis: \" + hyps[showid])\n",
    "    logger.info(\"example reference: \" + refs[showid])\n",
    "    \n",
    "    # show bleu results\n",
    "    logger.info(f\"validation loss:\\t{stats['loss']:.4f}\")\n",
    "    logger.info(stats[\"bleu\"].format())\n",
    "    return stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 保存和载入模型权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T13:37:37.190688Z",
     "iopub.status.busy": "2023-03-12T13:37:37.189232Z",
     "iopub.status.idle": "2023-03-12T13:37:37.209559Z",
     "shell.execute_reply": "2023-03-12T13:37:37.208131Z",
     "shell.execute_reply.started": "2023-03-12T13:37:37.190619Z"
    }
   },
   "outputs": [],
   "source": [
    "def validate_and_save(model, task, criterion, optimizer, epoch, save=True):   \n",
    "    stats = validate(model, task, criterion)\n",
    "    bleu = stats['bleu']\n",
    "    loss = stats['loss']\n",
    "    if save:\n",
    "        # 保存 epoch checkpoints\n",
    "        savedir = Path(config.savedir).absolute()\n",
    "        savedir.mkdir(parents=True, exist_ok=True)\n",
    "        \n",
    "        check = {\n",
    "            \"model\": model.state_dict(),\n",
    "            \"stats\": {\"bleu\": bleu.score, \"loss\": loss},\n",
    "            \"optim\": {\"step\": optimizer._step}\n",
    "        }\n",
    "        torch.save(check, savedir/f\"checkpoint{epoch}.pt\")\n",
    "        shutil.copy(savedir/f\"checkpoint{epoch}.pt\", savedir/f\"checkpoint_last.pt\")\n",
    "        logger.info(f\"saved epoch checkpoint: {savedir}/checkpoint{epoch}.pt\")\n",
    "    \n",
    "        # 保存 epoch 例子\n",
    "        with open(savedir/f\"samples{epoch}.{config.source_lang}-{config.target_lang}.txt\", \"w\") as f:\n",
    "            for s, h in zip(stats[\"srcs\"], stats[\"hyps\"]):\n",
    "                f.write(f\"{s}\\t{h}\\n\")\n",
    "\n",
    "        # 获取验证中最佳的 bleu    \n",
    "        if getattr(validate_and_save, \"best_bleu\", 0) < bleu.score:\n",
    "            validate_and_save.best_bleu = bleu.score\n",
    "            torch.save(check, savedir/f\"checkpoint_best.pt\")\n",
    "            \n",
    "        del_file = savedir / f\"checkpoint{epoch - config.keep_last_epochs}.pt\"\n",
    "        if del_file.exists():\n",
    "            del_file.unlink()\n",
    "    \n",
    "    return stats\n",
    "\n",
    "def try_load_checkpoint(model, optimizer=None, name=None):\n",
    "    name = name if name else \"checkpoint_last.pt\"\n",
    "    checkpath = Path(config.savedir)/name\n",
    "    if checkpath.exists():\n",
    "        check = torch.load(checkpath)\n",
    "        model.load_state_dict(check[\"model\"])\n",
    "        stats = check[\"stats\"]\n",
    "        step = \"unknown\"\n",
    "        if optimizer != None:\n",
    "            optimizer._step = step = check[\"optim\"][\"step\"]\n",
    "        logger.info(f\"loaded checkpoint {checkpath}: step={step} loss={stats['loss']} bleu={stats['bleu']}\")\n",
    "    else:\n",
    "        logger.info(f\"no checkpoints found at {checkpath}!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  &#x1F4CC; 开始训练！\n",
    "## 训练循环"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T13:37:46.147760Z",
     "iopub.status.busy": "2023-03-12T13:37:46.147268Z",
     "iopub.status.idle": "2023-03-12T13:37:46.156970Z",
     "shell.execute_reply": "2023-03-12T13:37:46.155414Z",
     "shell.execute_reply.started": "2023-03-12T13:37:46.147712Z"
    }
   },
   "outputs": [],
   "source": [
    "model = model.to(device=device)\n",
    "criterion = criterion.to(device=device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T13:37:46.798936Z",
     "iopub.status.busy": "2023-03-12T13:37:46.797754Z",
     "iopub.status.idle": "2023-03-12T13:37:46.810223Z",
     "shell.execute_reply": "2023-03-12T13:37:46.808737Z",
     "shell.execute_reply.started": "2023-03-12T13:37:46.798875Z"
    }
   },
   "outputs": [],
   "source": [
    "logger.info(\"task: {}\".format(task.__class__.__name__))\n",
    "logger.info(\"encoder: {}\".format(model.encoder.__class__.__name__))\n",
    "logger.info(\"decoder: {}\".format(model.decoder.__class__.__name__))\n",
    "logger.info(\"criterion: {}\".format(criterion.__class__.__name__))\n",
    "logger.info(\"optimizer: {}\".format(optimizer.__class__.__name__))\n",
    "logger.info(\n",
    "    \"num. model params: {:,} (num. trained: {:,})\".format(\n",
    "        sum(p.numel() for p in model.parameters()),\n",
    "        sum(p.numel() for p in model.parameters() if p.requires_grad),\n",
    "    )\n",
    ")\n",
    "logger.info(f\"max tokens per batch = {config.max_tokens}, accumulate steps = {config.accum_steps}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T13:37:47.551139Z",
     "iopub.status.busy": "2023-03-12T13:37:47.549778Z",
     "iopub.status.idle": "2023-03-12T14:16:06.192844Z",
     "shell.execute_reply": "2023-03-12T14:16:06.191289Z",
     "shell.execute_reply.started": "2023-03-12T13:37:47.551087Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train epoch 10:   3%|▎         | 20/791 [00:08<06:08,  2.09it/s, loss=4.39]Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7f2f34ed24d0>\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py\", line 1358, in __del__\n",
      "    self._shutdown_workers()\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py\", line 1341, in _shutdown_workers\n",
      "    if w.is_alive():\n",
      "  File \"/opt/conda/lib/python3.7/multiprocessing/process.py\", line 151, in is_alive\n",
      "    assert self._parent_pid == os.getpid(), 'can only test a child process'\n",
      "AssertionError: can only test a child process\n",
      "train epoch 10:   6%|▌         | 47/791 [00:20<04:40,  2.65it/s, loss=3.98]Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7f2f34ed24d0>\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py\", line 1358, in __del__\n",
      "    self._shutdown_workers()\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py\", line 1341, in _shutdown_workers\n",
      "    if w.is_alive():\n",
      "  File \"/opt/conda/lib/python3.7/multiprocessing/process.py\", line 151, in is_alive\n",
      "    assert self._parent_pid == os.getpid(), 'can only test a child process'\n",
      "AssertionError: can only test a child process\n",
      "train epoch 10: 100%|██████████| 791/791 [05:50<00:00,  2.26it/s, loss=4.07]\n",
      "train epoch 11: 100%|██████████| 791/791 [05:56<00:00,  2.22it/s, loss=4.22]\n",
      "train epoch 12: 100%|██████████| 791/791 [05:58<00:00,  2.21it/s, loss=3.93]\n",
      "train epoch 13: 100%|██████████| 791/791 [05:59<00:00,  2.20it/s, loss=4.32]\n",
      "train epoch 14: 100%|██████████| 791/791 [05:59<00:00,  2.20it/s, loss=4.52]\n",
      "train epoch 15: 100%|██████████| 791/791 [05:56<00:00,  2.22it/s, loss=4.01]\n",
      "                                                                            \r"
     ]
    }
   ],
   "source": [
    "epoch_itr = load_data_iterator(task, \"train\", config.start_epoch, config.max_tokens, config.num_workers)\n",
    "try_load_checkpoint(model, optimizer, name=config.resume)\n",
    "while epoch_itr.next_epoch_idx <= config.max_epoch:\n",
    "    # train for one epoch\n",
    "    train_one_epoch(epoch_itr, model, task, criterion, optimizer, config.accum_steps)\n",
    "    stats = validate_and_save(model, task, criterion, optimizer, epoch=epoch_itr.epoch)\n",
    "    logger.info(\"end of epoch {}\".format(epoch_itr.epoch))    \n",
    "    epoch_itr = load_data_iterator(task, \"train\", epoch_itr.next_epoch_idx, config.max_tokens, config.num_workers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 提交预测结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T14:16:06.198013Z",
     "iopub.status.busy": "2023-03-12T14:16:06.197074Z",
     "iopub.status.idle": "2023-03-12T14:16:12.666684Z",
     "shell.execute_reply": "2023-03-12T14:16:12.664958Z",
     "shell.execute_reply.started": "2023-03-12T14:16:06.197949Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Namespace(checkpoint_upper_bound=None, inputs=['./checkpoints/rnn'], num_epoch_checkpoints=5, num_update_checkpoints=None, output='./checkpoints/rnn/avg_last_5_checkpoint.pt')\n",
      "averaging checkpoints:  ['./checkpoints/rnn/checkpoint15.pt', './checkpoints/rnn/checkpoint14.pt', './checkpoints/rnn/checkpoint13.pt', './checkpoints/rnn/checkpoint12.pt', './checkpoints/rnn/checkpoint11.pt']\n",
      "Finished writing averaged checkpoint to ./checkpoints/rnn/avg_last_5_checkpoint.pt\n"
     ]
    }
   ],
   "source": [
    "# 平均几个检查点checkpoints可以产生与ensemble类似的效果\n",
    "checkdir=config.savedir\n",
    "!python ./fairseq/scripts/average_checkpoints.py \\\n",
    "--inputs {checkdir} \\\n",
    "--num-epoch-checkpoints 5 \\\n",
    "--output {checkdir}/avg_last_5_checkpoint.pt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 确认用于生成提交的模型权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # checkpoint_last.pt : 最后个 epoch的保存点\n",
    "# # checkpoint_best.pt : 验证集中 bleu 最高的保存点\n",
    "# # avg_last_5_checkpoint.pt:　过去5个epoched的平均值\n",
    "# try_load_checkpoint(model, name=\"avg_last_5_checkpoint.pt\")\n",
    "# validate(model, task, criterion, log_to_wandb=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成预测结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T14:16:12.670206Z",
     "iopub.status.busy": "2023-03-12T14:16:12.669277Z",
     "iopub.status.idle": "2023-03-12T14:16:12.684340Z",
     "shell.execute_reply": "2023-03-12T14:16:12.682855Z",
     "shell.execute_reply.started": "2023-03-12T14:16:12.670141Z"
    }
   },
   "outputs": [],
   "source": [
    "def generate_prediction(model, task, split=\"test\", outfile=\"./prediction.txt\"):    \n",
    "    task.load_dataset(split=split, epoch=1)\n",
    "    itr = load_data_iterator(task, split, 1, config.max_tokens, config.num_workers).next_epoch_itr(shuffle=False)\n",
    "    \n",
    "    idxs = []\n",
    "    hyps = []\n",
    "\n",
    "    model.eval()\n",
    "    progress = tqdm(itr, desc=f\"prediction\")\n",
    "    with torch.no_grad():\n",
    "        for i, sample in enumerate(progress):\n",
    "            # 验证集损失\n",
    "            sample = utils.move_to_cuda(sample, device=device)\n",
    "\n",
    "            # 模型推理\n",
    "            s, h, r = inference_step(sample, model)\n",
    "            \n",
    "            hyps.extend(h)\n",
    "            idxs.extend(list(sample['id']))\n",
    "            \n",
    "    # 根据预处理前的顺序排序\n",
    "    hyps = [x for _,x in sorted(zip(idxs,hyps))]\n",
    "    \n",
    "    with open(outfile, \"w\") as f:\n",
    "        for h in hyps:\n",
    "            f.write(h+\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T14:16:26.900992Z",
     "iopub.status.busy": "2023-03-12T14:16:26.899988Z",
     "iopub.status.idle": "2023-03-12T14:16:48.626140Z",
     "shell.execute_reply": "2023-03-12T14:16:48.624049Z",
     "shell.execute_reply.started": "2023-03-12T14:16:26.900949Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "prediction: 100%|██████████| 17/17 [00:19<00:00,  1.16s/it]\n"
     ]
    }
   ],
   "source": [
    "generate_prediction(model, task)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-12T14:18:51.796002Z",
     "iopub.status.busy": "2023-03-12T14:18:51.795420Z",
     "iopub.status.idle": "2023-03-12T14:18:54.445287Z",
     "shell.execute_reply": "2023-03-12T14:18:54.443661Z",
     "shell.execute_reply.started": "2023-03-12T14:18:51.795951Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "▁so ▁, ▁the ▁first ▁thing ▁that ▁we ▁did ▁is ▁we ▁gave ▁a d ul t s ▁a ▁ ver s ion ▁of ▁the ▁pi rate ▁problem ▁that ▁we ▁gave ▁to ▁the ▁kids ▁.\n",
      "▁that ' s ▁the ▁only ▁ex pla n ation ▁. ▁because ▁why ▁would ▁you ▁have ▁to ▁tell ▁people ▁that ▁something ▁was ▁actually ▁ho t ▁?\n",
      "所以 , 我們做的第一件事是 , 我們給孩子們帶來了許多問題 。\n",
      "這是唯一解釋 , 因為為什麼你必須告訴人們 , 很熱衷 ?\n"
     ]
    }
   ],
   "source": [
    "!head -n 2 /kaggle/working/DATA/rawdata/ted2020/test.en\n",
    "!head -n 2 prediction.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 反向翻译`Back-translation`\n",
    "## 训练反向翻译模型\n",
    "1. 在**config中切换source_lang和target_lang** \n",
    "2. 更改**config**中的savedir（例如“./checkpoints/transformer back”）\n",
    "3. 训练模型\n",
    "\n",
    "## 使用反向模型生成合成数据\n",
    "### 下载 monolingual 数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mono_dataset_name = 'mono'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mono_prefix = Path(data_dir).absolute() / mono_dataset_name\n",
    "mono_prefix.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "urls = (\n",
    "    \"https://github.com/yuhsinchan/ML2022-HW5Dataset/releases/download/v1.0.2/ted_zh_corpus.deduped.gz\"\n",
    ")\n",
    "file_names = (\n",
    "    'ted_zh_corpus.deduped.gz',\n",
    ")\n",
    "\n",
    "for u, f in zip(urls, file_names):\n",
    "    path = mono_prefix/f\n",
    "    if not path.exists():\n",
    "        else:\n",
    "            !wget {u} -O {path}\n",
    "    else:\n",
    "        print(f'{f} is exist, skip downloading')\n",
    "    if path.suffix == \".tgz\":\n",
    "        !tar -xvf {path} -C {prefix}\n",
    "    elif path.suffix == \".zip\":\n",
    "        !unzip -o {path} -d {prefix}\n",
    "    elif path.suffix == \".gz\":\n",
    "        !gzip -fkd {path}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=darkred><b>***TODO***: 数据清洗(clean corpus)</font></b>\n",
    "\n",
    "1. 删除过长或过短的句子\n",
    "2. 统一标点符号\n",
    "\n",
    "提示: 你可以使用上述定义的 `clean_s()` 来完成该操作 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  <font color=darkred><b>***TODO***: 子串(Subword Units)</font></b>\n",
    "\n",
    "使用反向模型的spm模型将数据标记为子字单位\n",
    "\n",
    "提示: spm 模型本地位置 DATA/raw-data/\\[dataset\\]/spm\\[vocab_num\\].model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 二进制化\n",
    "\n",
    "使用 fairseq 将数据转成二进制"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "binpath = Path('./DATA/data-bin', mono_dataset_name)\n",
    "src_dict_file = './DATA/data-bin/ted2020/dict.en.txt'\n",
    "tgt_dict_file = src_dict_file\n",
    "monopref = str(mono_prefix/\"mono.tok\") # whatever filepath you get after applying subword tokenization\n",
    "if binpath.exists():\n",
    "    print(binpath, \"exists, will not overwrite!\")\n",
    "else:\n",
    "    !python -m fairseq_cli.preprocess\\\n",
    "        --source-lang 'zh'\\\n",
    "        --target-lang 'en'\\\n",
    "        --trainpref {monopref}\\\n",
    "        --destdir {binpath}\\\n",
    "        --srcdict {src_dict_file}\\\n",
    "        --tgtdict {tgt_dict_file}\\\n",
    "        --workers 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=darkred><b>***TODO***: 使用反向模型生成合成数据</font></b> \n",
    "\n",
    "将二进制`monolingual data`添加到原始数据目录，并使用“split_name”命名\n",
    "比如： . ./DATA/data-bin/ted2020/\\[split_name\\].zh-en.\\[\"en\", \"zh\"\\].\\[\"bin\", \"idx\"\\]\n",
    "\n",
    "然后，你可以使用 `generate_prediction(model, task, split=\"split_name\")` 去生成翻译预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp ./DATA/data-bin/mono/train.zh-en.zh.bin ./DATA/data-bin/ted2020/mono.zh-en.zh.bin\n",
    "!cp ./DATA/data-bin/mono/train.zh-en.zh.idx ./DATA/data-bin/ted2020/mono.zh-en.zh.idx\n",
    "!cp ./DATA/data-bin/mono/train.zh-en.en.bin ./DATA/data-bin/ted2020/mono.zh-en.en.bin\n",
    "!cp ./DATA/data-bin/mono/train.zh-en.en.idx ./DATA/data-bin/ted2020/mono.zh-en.en.idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 参考\n",
    "\n",
    "1. <a name=ott2019fairseq></a>Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., ... & Auli, M. (2019, June). fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) (pp. 48-53).\n",
    "2. <a name=vaswani2017></a>Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017, December). Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (pp. 6000-6010).\n",
    "3. <a name=reimers-2020-multilingual-sentence-bert></a>Reimers, N., & Gurevych, I. (2020, November). Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 4512-4525).\n",
    "4. <a name=tiedemann2012parallel></a>Tiedemann, J. (2012, May). Parallel Data, Tools and Interfaces in OPUS. In Lrec (Vol. 2012, pp. 2214-2218).\n",
    "5. <a name=kudo-richardson-2018-sentencepiece></a>Kudo, T., & Richardson, J. (2018, November). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 66-71).\n",
    "6. <a name=sennrich-etal-2016-improving></a>Sennrich, R., Haddow, B., & Birch, A. (2016, August). Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 86-96).\n",
    "7. <a name=edunov-etal-2018-understanding></a>Edunov, S., Ott, M., Auli, M., & Grangier, D. (2018). Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 489-500).\n",
    "8. https://github.com/ajinkyakulkarni14/TED-Multilingual-Parallel-Corpus\n",
    "9. https://ithelp.ithome.com.tw/articles/10233122\n",
    "10. https://nlp.seas.harvard.edu/2018/04/03/attention.html\n",
    "11. https://colab.research.google.com/github/ga642381/ML2021-Spring/blob/main/HW05/HW05.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
